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Why  GAO  Did  This  Study 

For  more  than  a  decade,  the  Air  Force 
focused  its  training  on  supporting 
operations  in  the  Middle  East.  The  Air 
Force  has  established  goals  for  its 
combat  aircrews  to  conduct  training  for 
the  full  range  of  core  missions.  Both 
the  Senate  and  House  Reports 
accompanying  bills  for  the  FY  2016 
National  Defense  Authorization  Act 
included  a  provision  for  GAO  to  review 
the  Air  Force’s  training  plans. 

This  report  discusses  the  extent  to 
which  the  Air  Force  has  (1 )  determined 
requirements  to  train  combat  aircrews 
for  the  full  range  of  core  missions, 

(2)  met  annual  training  requirements 
for  combat  fighter  squadrons  across 
the  full  range  of  core  missions  and 
evaluated  the  effectiveness  of  this 
training,  and  (3)  established  virtual 
training  plans  that  include  desirable 
characteristics  of  a  comprehensive 
strategy.  GAO  reviewed  Air  Force 
training  requirements  and  plans  and 
interviewed  officials  with  a  non- 
generalizable  sample  of  units  based  on 
the  units’  range  of  core  missions. 

What  GAO  Recommends 

GAO  recommends  that  the  Air  Force 
(1 )  reassess  assumptions  for  annual 
training  requirements,  (2)  establish  and 
collect  data  on  desired  learning 
objectives  and  training  support 
elements  for  its  training  expectations, 
and  (3)  develop  a  risk-based 
investment  strategy  for  its  virtual 
training  plans.  DOD  concurred  with  the 
third  recommendation  and  did  not 
concur  with  the  first  and  second 
recommendations,  stating  that  existing 
initiatives  and  policies  address  these 
issues.  GAO  believes  the 
recommendations  remain  valid  for  the 
reasons  discussed  in  this  report. 

View  GAO-16-864.  For  more  information, 
contact  Cary  Russell  at  (202)  512-5431  or 
russellc@gao.gov. 


What  GAO  Found 

The  Air  Force  establishes  combat  aircrew  training  requirements  for  the  full  range 
of  core  missions  based  on  an  annual  process,  but  these  requirements  may  not 
reflect  current  and  emerging  training  needs,  because  the  Air  Force  has  not 
comprehensively  reassessed  the  assumptions  underlying  them.  Specifically, 
assumptions  about  the  total  annual  live-fly  sortie  requirements  by  aircraft,  the 
criteria  for  designating  aircrews  as  experienced  or  inexperienced,  and  the  mix 
between  live  and  simulator  training  have  remained  the  same  since  2012.  For 
example,  Air  Combat  Command  has  set  the  same  minimum  number  of  live-fly 
sortie  requirements  across  aircraft  platforms,  but  has  not  conducted  the  analysis 
needed  to  determine  if  requirements  should  differ  based  on  the  number  of  core 
missions  for  each  platform.  Reassessing  the  assumptions  underlying  annual 
training  requirements  would  better  position  the  Air  Force  to  meet  its  stated  goals 
for  its  forces  to  achieve  a  range  of  missions  for  current  and  emerging  threats. 

Combat  fighter  squadrons  were  generally  able  to  complete  mission  training 
requirements  for  ongoing  contingency  operations,  such  as  close  air  support  to 
ground  forces,  but  were  unable  to  meet  annual  training  requirements  across  the 
full  range  of  core  missions.  Further,  the  Air  Force  does  not  systematically 
evaluate  the  effectiveness  of  training  that  has  been  completed  against 
established  expectations.  Selected  unit  commanders  that  GAO  interviewed  cited 
four  common  factors  that  limited  their  ability  to  complete  training,  such  as  high 
deployment  rates,  and  other  factors  that  affected  the  training  that  aircrews  were 
able  to  accomplish.  However,  Air  Force  processes  used  to  record  and  monitor 
annual  training  do  not  include  a  systematic  evaluation  of  training  effectiveness 
against  expectations.  Specifically,  Air  Combat  Command  has  not  established  the 
desired  learning  objectives  or  training  support  elements  needed  to  accomplish 
training  expectations  and  does  not  collect  data  to  assess  effectiveness.  A  more 
consistent  basis  for  monitoring  results  is  critical  in  tracking  the  Air  Force’s 
progress  in  training  units  for  the  full  range  of  core  missions. 

Air  Force  plans  for  virtual  training  do  not  include  all  desirable  characteristics  of  a 
comprehensive  strategy,  such  as  a  risk-based  investment  strategy  or  a  time  line 
for  addressing  training  needs.  A  strategy  that  included  these  elements  would 
help  ensure  that  the  Air  Force’s  plans  addressed  its  capability  needs. 

This  is  a  public  version  of  a  sensitive  report  GAO  issued  in  August  2016.  It  omits 
sensitive  information  and  data  on  some  of  the  Air  Force’s  training  priorities, 
completion  of  annual  training  requirements  for  active-duty  fighter  squadrons,  and 
aircraft  maintenance  generation  capabilities. 
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Congressional  Committees 

For  more  than  a  decade,  the  Air  Force  focused  the  training  of  its  forces 
on  supporting  operations  in  the  Middle  East,  including  Iraq  and 
Afghanistan.  Commanders  established  training  requirements  that  they 
deemed  necessary  to  prepare  aircrews  to  conduct  missions  in  these 
locations — such  as  close  air  support  to  ground  forces — limiting  training 
opportunities  in  other  areas.  According  to  Air  Force  reports,  in  the  coming 
years,  the  Air  Force  will  confront  an  increasingly  complex  security 
environment  that  will  demand  a  wider  range  of  skill  sets  and  different 
capabilities  than  are  currently  being  employed.1  For  example,  aircrews 
may  be  called  upon  to  conduct  missions  that  require  freedom  of 
maneuver  in  highly-contested  air  spaces.  To  meet  these  emerging  needs, 
the  Air  Force  has  established  goals  for  its  fighter  aircrews  to  conduct 
training  for  a  full  range  of  core  missions.2 

Replicating  complex  threats — such  as  a  highly-contested  airspace — in  a 
live  training  environment  is  resource  intensive  and  can  be  constrained  by 
several  factors,  including  airspace  restrictions,  weather,  and  range 
capabilities,  among  others.  The  Air  Force  has  stated  that  by  integrating 
live,  virtual,  and  constructive  (LVC)  components  into  training  it  could 
overcome  some  of  the  limitations  of  training  in  a  live-only  environment  but 
notes  that  it  will  take  time  and  dedicated  resources  to  address 
technological  and  capacity  limitations.3  The  Air  Force  has  further  stated 


^ee,  for  example,  Secretary  of  the  Air  Force,  America’s  Air  Force:  A  Call  to  the  Future 
(July  2014);  U.S.  Air  Force,  USAF  Strategic  Master  Plan  (May  2015). 

2For  the  purposes  of  this  report,  we  use  the  phrase  “full  range  of  core  missions”  to 
describe  the  primary  and  secondary  missions  as  established  in  each  aircraft’s  Designed 
Operational  Capability  statement,  which  summarizes  specific  capabilities  that  an  Air  Force 
unit  can  be  called  on  to  provide. 

According  to  Air  Force  documents,  the  live  (L)  environment  is  defined  as  real  people 
operating  real  weapons  systems,  the  virtual  (V)  environment  is  defined  as  real  people 
operating  simulated  systems,  and  the  constructive  (C)  environment  is  identified  as 
software  models  and  code  that  are  used  to  improve  training  scenarios  with  computer¬ 
generated  entities — such  as  terrain,  threats,  aircraft,  people,  and  vehicles,  among  others. 
For  the  purposes  of  this  report,  we  refer  to  training  that  includes  a  simulator  as  virtual 
training. 
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that  the  integration  of  these  components  is  particularly  important  for  fifth- 
generation  aircraft,  such  as  the  F-35,  which  will  rely  more  heavily  on 
virtual  training  because  of  the  limitations  of  training  in  the  live 
environment  and  security  considerations  attributable  to  the  sensitive 
capabilities  of  these  aircraft. 

We  have  previously  reported  on  virtual  training  issues  for  the  Air  Force.  In 
2012,  we  assessed  the  Air  Force’s  virtual  training  efforts  in  response  to 
the  Secretary  of  Defense’s  efficiency  initiatives.4  We  found  that  the  Air 
Force  lacked  some  key  elements  of  an  overarching  organizational 
framework  that  were  needed  to  fully  integrate  virtual  training  into  overall 
training  goals  and  that  it  had  no  methodology  for  determining  the  costs  of 
its  virtual  training  enterprise.  We  recommended  that  the  Air  Force 
designate  an  entity  to  integrate  its  virtual  training  efforts,  develop  a 
strategy  to  align  virtual  training  initiatives  and  goals,  and  develop  a 
methodology  to  collect  cost  data  on  virtual  training.  In  response,  the  Air 
Force  has  taken  a  series  of  actions  to  improve  the  planning  and 
management  of  its  virtual  training  efforts.  For  example,  in  2012  the 
Secretary  of  the  Air  Force  issued  a  guidance  letter  that  restructured  the 
management  of  virtual  training  responsibilities  within  Headquarters  Air 
Force  and  designated  the  Air  Force  Agency  for  Modeling  and  Simulation 
as  the  lead  implementation  organization  for  LVC  operational  training 
integration.  According  to  Air  Force  officials,  the  Air  Force  is  also  in  the 
process  of  updating  guidance  on  the  management,  acquisition, 
modification,  and  modernization  of  simulator  and  other  training  devices. 
We  analyze  other  actions  the  Air  Force  has  taken  related  to  virtual 
training  later  in  this  report. 

Both  the  Senate  and  House  reports  accompanying  the  bills  for  the  2016 
National  Defense  Authorization  Act  included  provisions  for  us  to  review 
the  Air  Force’s  training  plans  and  requirements.5  This  report  discusses 
the  extent  to  which  the  Air  Force  has  (1)  determined  requirements  to  train 


4GAO,  Air  Force  Training:  Actions  Needed  to  Better  Manage  and  Determine  Costs  of 
Virtual  Training  Efforts,  GAO-12-727  (Washington,  D.C.:  July  19,  2012). 

5S.  Rep.  No.  114-49  and  H.  Rep.  No.  114-102.  Both  reports’  provisions  also  included 
review  of  the  Army’s  training  plans  and  requirements,  which  we  addressed  in  a  separate 
report.  See  GAO,  Army  Training:  Efforts  to  Adjust  Training  Requirements  Should 
Consider  the  Use  of  Virtual  Training  Devices,  GAO-16-636  (Washington,  D.C.:  Aug.  16, 
2016). 
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its  combat  aircrews  for  the  full  range  of  core  missions,  (2)  met  annual 
training  requirements  for  combat  fighter  squadrons  across  the  full  range 
of  core  missions  and  evaluated  the  effectiveness  of  this  training,  and  (3) 
established  virtual  training  plans  that  include  desirable  characteristics  of  a 
comprehensive  strategy. 

This  report  is  a  public  version  of  the  prior  sensitive  report  that  we  issued 
in  August  201 6. 6  The  Department  of  Defense  (DOD)  deemed  some  of  the 
information  in  the  prior  report  as  For  Official  Use  Only  (FOUO),  which 
must  be  protected  from  public  disclosure.  Therefore,  this  report  omits 
FOUO  information  and  data  on  some  of  the  Air  Force’s  training  priorities, 
completion  of  annual  training  requirements  for  active-duty  fighter 
squadrons,  and  aircraft  maintenance  generation  capabilities.  Although  the 
information  provided  in  this  report  is  more  limited  in  scope,  it  addresses 
the  same  objectives  as  the  sensitive  report.  Also,  the  methodology  used 
for  both  reports  is  the  same. 

We  focused  our  review  on  the  annual  requirements  for  continuation 
training  as  established  in  the  Ready  Aircrew  Program7  for  aircrews 
assigned  to  aircraft  within  the  active  component  of  the  combat  air  forces.8 
The  aircraft  included  in  our  review  were  the  A-10,  F-15,  F-16,  F-22,  and 
F-35.  To  determine  the  extent  to  which  the  Air  Force  has  determined 
requirements  to  train  its  combat  aircrews  for  the  full  range  of  core 
missions,  we  analyzed  Air  Force  documents  to  identify  the  Air  Force’s 
current  process  for  developing  annual  training  requirements  and  reviewed 
changes  in  these  requirements  for  fiscal  years  2012  through  2016.  We 
evaluated  these  processes  against  DOD  and  Air  Force  guidance  that 


6GAO,  Air  Force  Training:  Further  Analysis  and  Planning  Needed  to  Improve 
Effectiveness,  GAO-1 6-635SU  (Washington,  D.C.:  Aug.  16,  2016). 

7The  Ready  Aircrew  Program  establishes  the  minimum  number  of  live  training  events  and 
simulator  missions  that  aircrews  must  complete  during  the  annual  training  cycle. 

8ln  this  report,  we  refer  to  continuation  training  as  annual  training  requirements.  The 
Ready  Aircrew  Program  establishes  the  annual  training  requirements  aircrews  must 
accomplish  to  sustain  combat  mission  readiness,  as  discussed  later  in  this  report. 
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prioritizes  and  establishes  training  requirements  and  leading  practices 
that  we  have  identified  for  managing  strategic  training.9 

To  determine  the  extent  to  which  the  Air  Force  has  met  annual  training 
requirements  for  combat  fighter  squadrons  across  the  full  range  of  core 
missions  and  evaluated  the  effectiveness  of  this  training,  we  analyzed 
training  completion  data  from  fiscal  years  2012  through  2015  for  the 
combat  airforces.  We  performed  data  reliability  procedures  on 
information  included  in  Ready  Aircrew  Program  reports  by  comparing  the 
data  against  related  documentation  and  interviewing  knowledgeable 
officials  on  controls  over  reporting  systems  and  determined  that  the  data 
presented  in  our  findings  were  sufficiently  reliable  for  the  purposes  of  this 
report.  We  reviewed  Air  Force  documentation  that  describes  training 
challenges  and  interviewed  Air  Force  officials,  including  wing  and 
squadron  commanders  at  selected  units,  to  discuss  any  factors  that 
limited  the  ability  of  aircrews  to  complete  training  for  the  full  range  of  core 
missions.  We  selected  a  non-generalizable  sample  of  units  to  speak  with 
based  on  our  analysis  of  training  completion  data  and  the  range  of  units’ 
core  missions.  We  evaluated  the  Air  Force’s  process  for  assessing  the 
effectiveness  of  its  annual  training  against  leading  practices  that  we  have 
identified  for  managing  strategic  training  and  the  Ready  Aircrew  Program 
tasking  memorandums  that  specify  the  requirements  for  annual 
continuation  training  for  personnel  assigned  to  combat  units.10 

To  determine  the  extent  to  which  the  Air  Force  has  established  virtual 
training  plans  that  include  desirable  characteristics  of  comprehensive 
strategies,  we  reviewed  planning  documents  established  by  the  Air  Force 
to  guide  its  virtual  training  efforts  and  evaluated  those  actions  against  our 


9GAO,  Human  Capital:  A  Guide  for  Assessing  Strategic  Training  and  Development  Efforts 
in  the  Federal  Government,  GAO-04-546G  (Washington  D.C.:  Mar.  1,  2004).  This  guide 
introduces  a  framework,  consisting  of  a  set  of  principles  and  key  questions  that  federal 
agencies  can  use  to  ensure  that  their  training  and  development  investments  are  targeted 
strategically.  Information  in  this  guide  was  developed  through  consultations  with 
government  officials  and  experts  in  the  private  sector,  academia,  and  nonprofit 
organizations;  examinations  of  laws  and  regulations  related  to  training  and  development  in 
the  federal  government;  and  reviews  of  the  sizeable  body  of  literature  on  training  and 
development  issues,  including  previous  GAO  products  on  a  range  of  human  capital  topics. 

10GAO-04-546G. 
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prior  work  on  desirable  characteristics  of  national  strategies.* 11  We 
determined  whether  these  actions  addressed,  partially  addressed,  or  did 
not  address  the  desirable  characteristics  of  strategy  documents,  based  on 
our  assessment  of  whether  the  planning  documents  explicitly  cited  all 
elements  of  a  characteristic  and  the  level  of  specificity  and  detail 
included.  Further  details  on  our  scope  and  methodology  can  be  found  in 
appendix  I. 

We  conducted  this  performance  audit  from  June  2015  to  September  2016 
in  accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 


Background 


The  Secretary  of  the  Air  Force  is  responsible  for  recruiting,  organizing, 
training,  and  equipping  forces  in  order  to  meet  the  current  and  future 
operational  requirements  of  DOD’s  combatant  commands.12  The  Air 
Force  is  organized  into  10  major  commands  that  are  assigned  a  major 
part  of  the  Air  Force  mission.13  Numbered  Air  Forces  fall  directly  under 


11  We  assessed  planning  documents  established  by  the  Air  Force  to  guide  its  virtual 
training  efforts  against  criteria  that  we  developed  for  national  strategies.  These  criteria  are 
cited  in  GAO,  Combating  Terrorism:  Evaluation  of  Selected  Characteristics  in  National 
Strategies  Related  to  Terrorism,  GAO-04-408T  (Washington,  D.C.:  Feb.  3,  2004).  Given 
the  size  and  complexity  of  the  Air  Force’s  virtual  training  efforts,  we  concluded  that  a 
comprehensive  strategy  to  guide  these  efforts  would  be  similar  in  scope  and  would  need 
to  include  characteristics  similar  to  those  of  a  national  strategy. 

12The  Air  Force  supports  DOD’s  six  geographic  combatant  commands  and  three 
functional  combatant  commands.  The  six  combatant  commands  are  U.S.  Africa 
Command,  U.S.  Central  Command,  U.S.  European  Command,  U.S.  Northern  Command, 
U.S.  Pacific  Command,  and  U.S.  Southern  Command.  The  three  functional  combatant 
commands  are  U.S.  Special  Operations  Command  for  special  operations  forces,  U.S. 
Strategic  Command  for  space  and  nuclear  forces,  and  U.S.  Transportation  Command  for 
air,  land,  and  sealift  forces. 

13The  Air  Force’s  major  commands  are  Air  Combat  Command,  Air  Education  and  Training 
Command,  Air  Force  Global  Strike  Command,  Air  Force  Materiel  Command,  Air  Force 
Reserve  Command,  Air  Force  Space  Command,  Air  Force  Special  Operations  Command, 
Air  Mobility  Command,  U.S.  Air  Forces  in  Europe — Air  Forces  Africa,  and  Pacific  Air 
Forces. 
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the  major  commands  and  are  focused  on  ensuring  the  readiness  of 
assigned  forces.  For  example,  Pacific  Air  Forces  contains  three 
numbered  air  forces  that  are  located  in  South  Korea,  Japan,  and  Alaska. 
They  provide  operational  leadership  and  supervision  and  are  assigned 
subordinate  units,  such  as  wings,  groups,  and  squadrons.  The 
component  numbered  Air  Force  is  a  specialized  category  of  numbered  Air 
Force  that  is  structured  to  perform  an  operational  and  warfighting  mission 
in  support  of  a  combatant  command.  Air  Force  units,  such  as  an 
operational  flying  squadron,  are  designed  to  provide  a  specific  set  of 
mission  capabilities  to  meet  anticipated  operational  requirements.  See 
figure  1  for  general  characteristics  of  the  combat  aircraft  platforms 
included  in  our  review.  Each  combat  unit  has  a  Designed  Operational 
Capability  statement  that  it  uses  for  reporting  readiness  data  and 
includes,  among  other  things,  the  full  range  of  core  missions  that  the  unit 
can  be  called  on  to  provide.  Air  Combat  Command  has  responsibility  for 
developing  and  managing  aircrew  training  requirements  to  meet  the  full 
range  of  core  missions. 
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Figure  1:  Characteristics  of  Selected  Fourth- and  Fifth-Generation  Combat  Aircraft  Platforms 
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Source:  Air  Forces  (data);  Defense  Video  and  Imagery  System  (photos).  |  GAO-16-864 

aPrimary  and  secondary  missions  listed  are  for  the  F-16  Block  50  aircraft. 
bThere  were  no  operational  F-35  squadrons  in  fiscal  year  2015. 


Throughout  their  careers,  Air  Force  personnel  designated  as  aircrews 
participate  in  several  different  types  of  training.  When  they  first  enter  the 
Air  Force,  they  receive  basic  military  training.  After  basic  military  training, 
aircrews  receive  additional  training,  including  undergraduate  training  by 
the  Air  Force’s  Air  Education  and  Training  Command.  Next,  aircrews 
receive  initial  qualification  training  that  is  focused  on  their  career 
specialties  and  is  provided  by  the  Air  Education  and  Training  Command 
for  the  F-1 6,  F-1 5C,  and  F-35  platforms  and  by  Air  Combat  Command  for 
the  A-10,  F-15E,  and  F-22  platforms.  After  aircrews  complete  initial 
qualification  training,  they  are  assigned  to  a  unit,  where  they  complete 
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mission  qualification  training — additional  training  on  the  skills  specific  to 
the  unit’s  capabilities.  Air  Force  Instruction  (AFI)  11-202  is  the 
foundational  aircrew  training  instruction  that  guides  training  programs  for 
all  Air  Force  flying  training.14  There  is  an  AFI  11-2,  Volume  1  for  each 
aircraft  platform  that  specifies  the  training  requirements  for  aircrews  who 
are  assigned  to  units  that  operate  that  platform.  For  example,  F-22  units 
would  use  AFI  1 1-2F-22A,  Volume  1 .  In  addition  to  AFI  11-202,  tasking 
memorandums  for  the  Ready  Aircrew  Program  specify  annual 
continuation  training  for  personnel  assigned  to  combat  units.  Specifically, 
Ready  Aircrew  Program  tasking  memorandums  establish  the  minimum 
number  of  live  training  events  (“sorties”)  and  simulator  missions  (or  virtual 
training)  that  aircrews  must  complete  during  the  annual  training  cycle  to 
be  considered  combat  mission  ready  or  basic  mission  capable  for  the 
assigned  aircraft.15  Finally,  aircrews  may  complete  upgrade  training  to 
qualify  for  positions  such  as  flight  leader,  instructor  pilot,  or  mission 
commander. 

“Combat  mission  ready”  is  a  squadron  commander  certification  based  on 
the  extent  to  which  aircrews  or  units  have  completed  and  sustained 
required  training.  Combat  mission  ready  status  is  maintained  by 
completing  the  annual  continuation  training  requirements  as  prescribed 
by  the  unit’s  AFI  11-2,  Volume  1 ,  and  the  Ready  Aircrew  Program  tasking 
memorandums.  These  requirements  include  completing  a  specific 
quantity  of  relatively  advanced  individual  and  unit  training  events  within  a 
given  time  period.  For  example,  experienced  F-22  aircrews  must  perform 
11  defensive  counter  air  training  sorties  in  a  12-month  period.  In  addition, 
aircrews  must  achieve  and  maintain  “currency”  by  completing  certain 
required  training  events  deemed  critical  to  safety  of  flight,  such  as 
landings,  low  altitude  flight,  and  formation  take-offs.  For  example, 
experienced  F-22  aircrews  are  to  land  the  aircraft  every  45  days  or  else 
they  would  lose  landing  currency.  Currency  events  are  typically  achieved 
by  completing  the  annual  training  requirements. 


14Air  Force  Instruction  11-202,  Volume  1 ,  Flying  Operations:  Aircrew  Training  (Nov.  22, 
2010). 

15For  the  purposes  of  this  report,  we  focus  on  combat  mission  ready  requirements — the 
training  requirements  that  indicate  that  aircrews  are  qualified,  current,  and  proficient  in  all 
of  the  primary  missions  tasked  to  their  assigned  combat  unit.  In  contrast,  basic  mission 
capable  aircrews  are  familiar  with  all  the  primary  missions  tasked  to  their  assigned  or 
attached  unit. 
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Most  Air  Force  combat  units  report  the  status  of  their  unit  training  through 
the  Air  Combat  Command’s  Ready  Aircrew  Program.  The  Ready  Aircrew 
Program  establishes  the  minimum  number  of  sorties  and  simulator 
missions  that  aircrews  must  accomplish  to  sustain  combat  mission 
readiness.  These  sorties  and  simulator  missions  are  aligned  with  the 
unit’s  primary  missions,  for  which  the  units  must  maintain  “proficiency,” 
and  secondary  missions,  for  which  they  must  maintain  “familiarity.”16  For 
example,  the  fiscal  year  201 6  tasking  memorandum  for  the  F-1 5E  aircraft 
specifies  that  experienced  aircrews  are  required  to  complete  a  minimum 
of  96  annual  live  sorties  (8  per  month)  to  be  considered  combat  mission 
ready,  of  which  49  are  flown  to  fulfill  primary  mission  requirements. 
Similarly,  inexperienced  aircrews  are  to  complete  108  annual  sorties  (9 
per  month),  of  which  54  are  flown  to  fulfill  primary  mission  requirements. 
Both  experienced  and  inexperienced  aircrews  are  required  to  complete 
36  annual  simulator  missions  (3  simulator  missions  per  month)  to  be 
considered  combat  mission  ready.  Units  are  required  to  submit  an  annual 
end-of-cycle  training  report  covering  the  12-month  training  cycle.  These 
reports  include  information  on  the  percentage  of  training  accomplished  by 
the  unit  against  the  training  requirements  for  its  primary  and  secondary 
missions,  as  well  as  any  factors  that  limit  the  unit’s  ability  to  meet  its 
training  requirements,  such  as  the  availability  of  personnel  or  aircraft. 

Virtual  training  uses  aircraft  simulators  that  replicate  the  live  environment 
for  various  missions  and  events.  Aircraft  simulators  can  be  standalone  or 
linked  to  information  networks  that  bring  together  geographically- 
separated  simulators  in  order  to  conduct  training.  Training  missions  that 
connect  geographically-separated  simulators  are  referred  to  as  distributed 
mission  operations.  Distributed  mission  operations  utilize  the  integration 
of  virtual  (e.g.,  a  person  training  in  a  simulator)  and  constructive  (e.g., 
computer-generated)  elements  to  train  aircrews.  To  date,  Air  Force  virtual 
training  efforts  have  focused  mainly  on  the  integration  of  virtual  and 
constructive  aspects  of  distributed  training.  However,  the  Air  Force’s  goal 
is  to  integrate  live  training  with  distributed  mission  operations,  which  is 
referred  to  as  Live,  Virtual,  and  Constructive  (LVC)  operational  training. 


16Proficiency  means  the  aircrew  is  ready  to  fly  the  mission  in  combat.  Familiarity  is  limited 
experience  with  a  mission  that  can  be  brought  up  to  proficiency  with  60  to  90  days  of 
focused  training. 
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The  Air  Force’s 
Process  for 
Developing  Combat 
Aircrew  Annual 
Training 

Requirements  for  the 
Full  Range  of  Core 
Missions  Is  Based  on 
Dated  Assumptions 


The  Air  Force  establishes  combat  aircrew  training  requirements  for  the 
full  range  of  core  missions  based  on  an  annual  process,  but  these 
requirements  may  not  reflect  current  and  emerging  training  needs, 
because  the  Air  Force  has  not  comprehensively  reassessed  the 
assumptions  underlying  them.  The  Air  Force  process  to  develop  annual 
training  requirements  for  the  aircrews  of  each  type  of  combat  aircraft 
includes  input  from  the  component  numbered  Air  Forces  and  concludes 
with  an  annual  training  review  board  that  brings  together  stakeholders. 
However,  these  requirements  may  not  reflect  current  and  emerging 
training  needs,  because  the  Air  Force  has  not  comprehensively 
reassessed  the  assumptions  underlying  the  total  number  of  annual  live-fly 
training  sorties  by  aircraft,  the  criteria  for  designating  aircrews  as 
experienced  or  inexperienced,  and  the  mix  between  live  and  simulator 
training  since  2012. 


Air  Combat  Command 
Has  a  Process  to  Develop 
Annual  Training 
Requirements  for  the  Full 
Range  of  Core  Missions 


The  Air  Force  has  established  a  process  to  develop  annual  training 
requirements  for  the  aircrews  of  each  type  of  combat  aircraft  for  the  full 
range  of  core  missions.  This  process  begins  with  identifying  the  capability 
needs  of  the  component  numbered  Air  Forces.  Every  two  years  the  Vice 
Commander  of  Air  Combat  Command  sends  a  letter  to  the  component 
numbered  Air  Forces  requesting  that  they  review  their  primary  and 
secondary  mission  lists  and  identify  their  priorities  for  Air  Force  units  by 
mission  and  type  of  combat  aircraft.  The  component  numbered  Air  Forces 
provide  their  responses  to  Air  Combat  Command,  describing  their 
expectations  and  priorities. 


Annually,  Air  Combat  Command  takes  the  expectations  and  priorities 
identified  by  the  component  numbered  Air  Forces  and  develops  the  draft 
Ready  Aircrew  Program  tasking  memorandums  that  set  the  annual 
training  requirements  for  each  aircraft.  Inputs  into  development  of  the 
draft  tasking  memorandums  include  subject-matter  experts’  assessments 
of  available  training  resources  and  the  mid-year  and  end-of-year  training 
and  readiness  reviews.  The  draft  Ready  Aircrew  Program  tasking 
memorandums  are  then  released  to  the  fighter  wings  to  obtain  feedback 
from  the  operations  group  commanders  on  their  units’  ability  to  execute 


Page  10 


GAO-16-864  Air  Force  Training 


the  training  requirements.17  Air  Combat  Command  officials  consolidate 
group  commander  responses  and  prepare  the  agenda  for  the  annual 
Realistic  Training  Review  Board.  The  review  board  consists  of  senior 
officers  who  meet  over  three  days  during  the  summer  to  review  the  draft 
Ready  Aircrew  Program  tasking  memorandums  and  make  their  final 
recommendations  to  Air  Combat  Command.  The  final  adjustments  are 
made  by  Air  Combat  Command,  and  the  Ready  Aircrew  Program  tasking 
memorandums  are  published  before  the  start  of  the  next  fiscal  year. 
Figure  2  summarizes  Air  Combat  Command’s  process  for  developing 
these  annual  training  requirements. 


Figure  2:  Air  Combat  Command’s  Process  for  Developing  Annual  Training 
Requirements 
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The  annual  training  requirements  are  used  to  estimate  the  Air  Force’s 
budget  request  for  the  flying-hour  program,  which  funds  live  training. 
These  hours  are  determined  using  the  Air  Force  Single  Flying  Hour 
Model,  which  provides  the  methodology  and  processes  that  the  major 


17The  annual  Weapons  and  Tactics  Conference — which  brings  together  warfighters  from 
across  the  combat  air  forces  to  discuss  current  issues,  look  at  future  issues,  and  provide 
solutions  for  joint  and  combined  employment  of  airpower — is  another  opportunity  for  units 
to  have  input  into  the  process  for  determining  annual  training  requirements. 
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commands  use  to  build  their  flying-hour  programs.  The  model  calculates 
the  flying  hours  needed  based  on  inputs  of  the  force  structure  (number  of 
aircraft  and  aircrew),  aircrew  data  (types  and  number  of  aircrew  that 
require  training),  and  live-fly  requirements  (established  in  the  Ready 
Aircrew  Program  tasking  memorandums  for  combat  air  forces). 
Headquarters  Air  Force  has  overall  program  management  responsibility 
for  the  flying-hour  program,  and  major  commands  are  responsible  for 
their  requirements  and  for  reporting  on  how  those  hours  are  used.  Air 
Combat  Command  performs  this  function  for  the  combat  air  forces.  Once 
flying  hours  are  allocated  to  units,  the  wing  commanders  manage  those 
hours  and  report  monthly  to  Headquarters  Air  Force  on  the  number  of 
hours  actually  flown. 


Annual  Training 
Requirements  Are  Based 
on  Dated  Assumptions 
and  May  Not  Reflect 
Current  and  Emerging 
Training  Needs 


The  Air  Force  strategy,  America’s  Air  Force:  A  Call  to  the  Future, 
discusses  the  need  to  ensure  a  full-spectrum-capable  force  that  can 
successfully  achieve  missions  to  address  a  broad  range  of  current  and 
emerging  threats.18  Further,  the  document  states  that  the  design  of 
training  should  move  to  achieve  sufficient  readiness  across  all  mission 
sets  while  utilizing  virtual  training  to  deliver  robust  and  realistic  training 
against  existing  and  emerging  threats.  On  an  annual  basis,  Air  Combat 
Command  makes  updates  to  its  training  plans  based  on  the  process 
discussed  above.  These  updates  can  include  changes  to  the  aircraft’s 
primary  and  secondary  missions,  the  allocation  of  sorties  among 
missions,  and  the  types  of  required  training  events,  among  other 
changes.  However,  based  on  our  review  of  annual  training  requirements 
for  combat  aircrews  from  2012  through  2016  and  discussions  with  Air 
Force  officials,  we  found  that  Air  Combat  Command  has  used  the  same 
underlying  assumptions  to  establish  its  annual  training  requirements. 
Specifically,  the  total  annual  live-fly  training  sorties  by  aircraft,  the  criteria 
for  designating  aircrews  as  experienced  or  inexperienced,  and  the  mix 
between  live  and  simulator  training  remained  the  same  during  this  time 
period  and  may  not  reflect  current  and  emerging  training  needs,  as 
discussed  below. 


Since  2012,  Air  Combat  Command  has  set  the  same  requirement  for  the 
minimum  number  of  live-fly  sorties  across  all  combat  aircraft  platforms, 


1  ft 

Secretary  of  the  Air  Force,  America’s  Air  Force:  A  Call  to  the  Future  (July  2014). 
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regardless  of  the  number  of  core  missions  assigned  to  each  platform.  For 
fiscal  year  2016,  for  example,  the  annual  requirement  for  live  training 
sorties  for  an  F-15C,  which  has  two  primary  missions  and  no  secondary 
missions,  is  the  same  as  the  annual  requirement  for  an  F-35,  which  has 
three  primary  missions  and  three  secondary  missions.  Officials  from  Air 
Combat  Command  stated  that  the  live-fly  sortie  requirements  are  based 
in  part  on  a  2010  Navy  analysis  of  the  number  of  sorties  needed  to 
perform  safe  flying  operations.  However,  two  of  four  wing  commanders 
we  spoke  with  noted  that  differences  in  the  number  of  core  missions 
should  be  considered  when  establishing  minimum  requirements  for  live 
training  sorties  by  platform,  because  of  variations  in  the  level  of 
complexity  of  each  aircraft  and  mission.  Air  Combat  Command  officials 
told  us  that  aircrews  flying  more  complex  combat  aircraft  may  require 
additional  training  but  stated  that  given  the  constrained  resource 
environment  that  DOD  and  other  federal  agencies  have  operated  under 
for  the  past  several  years,  it  would  be  difficult  to  increase  training 
requirements  and  the  flying  hours  required  to  meet  them,  even  in  the  face 
of  a  potential  need.  According  to  its  fiscal  year  2017  budget  request,  the 
Air  Force  capped  its  flying-hour  programs  at  a  level  it  determined  to  be 
executable  and  therefore  has  not  been  able  to  grow  readiness  in  the  full 
range  of  missions  over  previous  years.  The  budget  request  also  notes 
that  the  fiscal  year  201 7  request  is  intended  to  align  funding  for  the  flying- 
hour  program  to  maintain  current  readiness  levels  and  to  meet  its  highest 
priorities.  However,  Air  Combat  Command  officials  stated  that  the 
analysis  needed  to  determine  the  relative  differences  in  annual  live-fly 
training  requirements  based  on  the  complexity  of  various  aircraft  and 
missions  had  not  been  conducted. 

In  addition,  the  criteria  for  designating  experienced  and  inexperienced 
aircrews  are  based  on  assumptions  that  have  been  in  place  for  the  last  40 
years.  The  annual  training  requirements  differ  for  aircrews  designated  as 
experienced  and  those  designated  as  inexperienced.  For  example, 
inexperienced  aircrews  are  required  to  complete  one  more  training  sortie 
each  month  than  experienced  ones.  In  most  cases,  aircrews  are 
designated  as  experienced  if  they  have  flown  500  hours  in  their  primary 
aircraft.19  Certain  wing  and  squadron  officials  offered  that,  as  a  result  of 


19Simulator  training  accomplished  toward  annual  training  requirements  is  counted  as  an 
hour  to  determine  the  experienced  designation,  but  cannot  exceed  20  percent  of  the  total 
hours  when  determining  experience  level. 
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ongoing  deployments  to  the  Middle  East,  aircrews  can  achieve  500  hours 
in  their  primary  aircraft  on  their  first  deployment.  However,  because  the 
scope  of  missions  these  aircrews  conduct  while  deployed  is  narrow,  they 
are  not  necessarily  getting  training  and  experience  across  the  full  range 
of  core  missions.  As  a  result,  when  they  return  from  their  deployment, 
these  aircrews  can  be  designated  as  experienced,  and  therefore  they  will 
not  get  the  additional  training  that  they  may  need.  These  officials  also 
noted  that  aircrew  capability  within  the  squadron  may  suffer  and  there 
could  be  less  capacity  to  fill  squadron  positions  with  sufficiently  qualified 
aircrews. 

Further,  annual  training  requirements  since  fiscal  year  2012  have  set  the 
same  mix  of  live  and  simulator  requirements  across  most  of  the  platforms 
we  reviewed.  For  all  but  one  combat  aircraft  platform,  both  experienced 
and  inexperienced  aircrews  are  required  to  complete  36  simulator 
missions  per  year  to  be  considered  combat  mission  ready.20  These 
requirements  were  established  based  on  the  capacity  of  simulators  at  one 
installation  and  not  on  an  analysis  of  the  training  needs  of  aircrews. 
According  to  Air  Combat  Command  officials,  the  requirement  for  aircrews 
to  fly  36  simulator  missions  per  year  was  established  based  in  part  on  the 
simulator  capacity  at  Shaw  Air  Force  Base,  which  had  the  greatest 
demands  on  its  simulators  because  it  supports  three  active  fighter 
squadrons  and  an  Air  National  Guard  squadron.  However,  one  wing 
commander  and  two  squadron  commanders  told  us  that  simulator 
requirements  may  need  to  differ  among  aircraft,  depending  on  the  full 
range  of  core  missions  that  the  aircraft  is  expected  to  perform. 
Additionally,  Air  Combat  Command  officials  told  us  that  aircrews  of  fifth- 
generation  aircraft  may  require  additional  simulator  training.  In  fact,  the 
simulator  training  requirements  for  F-35  aircrews  include  one  more 
simulator  mission  per  month,  for  a  total  of  48  simulator  missions  per  year. 
However,  Air  Combat  Command  did  not  provide  any  analysis  that 
supports  the  need  for  an  additional  simulator  mission  for  the  F-35  or 
explain  why  the  F-22,  another  fifth-generation  aircraft,  would  not  also 
require  additional  simulator  training  for  its  aircrews  over  what  is  required 
for  fourth-generation  aircraft.  Based  on  our  review  of  fiscal  year  2015  data 
provided  by  Air  Combat  Command  on  available  simulator  hours  at 


20Twelve  of  the  36  simulator  missions  per  year  are  required  to  be  distributed  mission 
operations. 
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combat  air  forces  bases,  we  found  that  there  was  excess  simulator 
capacity  at  a  number  of  locations.  For  example,  F-15E  squadrons  at 
Mountain  Home  Air  Force  Base  had  an  annual  capacity  of  approximately 
6,000  hours  at  their  simulator  facility,  but  the  annual  simulator 
requirement  was  approximately  4,000  hours. 

According  to  our  prior  work  on  strategic  training,  training  plans  should  be 
designed  to  determine  the  skills  and  competencies  a  workforce  needs  to 
achieve  current,  emerging,  and  future  agency  missions  and  to  identify 
gaps — including  those  that  training  and  development  strategies  can  help 
address.  Periodic  reassessments  should  be  part  of  a  continual  effort  to 
evaluate  and  improve  the  agency’s  training  and  development  efforts,  and 
evidence  of  timely  changes  should  be  reflected  in  those  efforts.21  While 
Air  Combat  Command  makes  updates  to  the  annual  training  plans  each 
year,  it  has  not  comprehensively  reassessed  the  assumptions  underlying 
its  annual  training  requirements — including,  but  not  limited  to,  the  total 
annual  training  requirements  by  aircraft,  the  criteria  for  designating 
aircrews  as  experienced  or  inexperienced,  and  the  mix  between  live  and 
simulator  training — and  updated  these  new  assumptions  in  its  training 
plans. 

Officials  at  Air  Combat  Command  told  us  that  they  had  two  studies  under 
way  to  help  inform  the  needed  analysis  to  address  these  issues,  but  the 
studies  had  not  been  completed  at  the  time  of  our  review.  For  example, 

Air  Combat  Command  has  an  ongoing  study  evaluating  the  current 
requirement  that  aircrews  accomplish  500  flying  hours  in  order  to  be 
considered  experienced.  The  study  is  intended  to  explore  various  options 
for  the  F-22,  including  adjusting  the  current  500-hour  requirement  that  is 
used  to  determine  when  to  designate  an  aircrew  as  experienced.  Air 
Combat  Command  officials  stated  that  they  may  explore  applying  the 
results  of  this  study  to  other  combat  aircraft.  Additionally,  in  2016,  Air 
Combat  Command  issued  a  proposal  to  study  the  minimum,  safe,  and 
optimal  numbers  of  live-fly  and  simulator  training  sorties  aircrews  must 
complete  each  year.  The  study  is  intended  to  include  an  examination  of 
the  minimum  number  of  sorties  required  by  mission  type,  the  definition  of 
proficient  and  familiar,  and  the  mix  of  live  and  simulator  training  to  support 
each  mission  set,  among  other  areas.  According  to  officials  from  Air 
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Combat  Command,  the  results  of  the  study  are  expected  to  be  available 
in  2017. 

While  these  initiatives  are  a  positive  step,  it  is  too  early  to  determine  how 
the  results  of  the  two  studies  will  be  used  to  implement  changes  to  the  Air 
Force’s  annual  training  requirements  or  whether  the  scope  of  the  studies 
is  sufficient  to  address  all  of  the  assumptions  that  need  to  be  revisited. 
Without  fully  reassessing  the  assumptions  underlying  its  annual  training 
requirements,  the  Air  Force  cannot  be  certain  that  its  annual  training 
plans  are  aligned  with  its  stated  goals  to  ensure  a  full-spectrum-capable 
force  that  can  successfully  achieve  missions  across  a  broad  range  of 
current  and  emerging  threats. 


Combat  Fighter 
Squadrons  Are  Not 
Meeting  All  of  Their 
Annual  Training 
Requirements  across 
the  Full  Range  of 
Core  Missions,  and 
Completed  Training  Is 
Not  Being  Evaluated 
for  Effectiveness 
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Combat  Fighter 
Squadrons  Are  Not 
Meeting  All  of  Their  Annual 
Training  Requirements 
across  the  Full  Range  of 
Core  Missions 


Based  on  our  analysis  of  data  on  the  completion  of  annual  training,  we 
found  that  combat  fighter  squadrons  were  generally  able  to  complete 
mission  training  requirements  for  ongoing  contingency  operations,  such 
as  close  air  support  to  ground  forces,  but  were  unable  to  meet  annual 
training  requirements  across  the  full  range  of  core  missions.22 


Various  Factors  Affected 
the  Ability  of  Combat 
Fighter  Squadrons  to 
Complete  Annual  Training 
across  the  Full  Range  of 
Core  Missions 


Wing  and  squadron  commanders  cited  various  factors  that  have  limited 
the  ability  of  their  squadrons  to  complete  training  across  the  full  range  of 
core  missions.  We  analyzed  the  information  we  obtained  from  our 
interviews  with  four  wing  commanders  and  eight  squadron 
commanders — including  commanders  overseeing  aircrew  training  for  both 
fourth-  and  fifth-generation  aircraft — and  found  that  they  identified  similar 
training  limitations.23  They  cited  four  common  limitations:  the  maintenance 
unit’s  ability  to  provide  adequate  numbers  of  aircraft  for  training,  high 
deployment  rates,  adversary  air  tasking,  and  manpower  shortfalls  in  the 
squadrons. 


Seven  of  the  squadron  commanders  we  interviewed  told  us  that  their 
maintenance  unit’s  ability  to  provide  an  adequate  number  of  aircraft 
affected  their  squadrons’  ability  to  complete  their  annual  training 
requirements.24  These  commanders  stated  that  in  recent  years, 
maintenance  units  for  U.S. -based  squadrons  have  been  consistently 
manned  below  their  authorized  levels.  Air  Force  officials  told  us  these 


22The  number  and  type  of  primary  and  secondary  missions  can  differ  both  by  aircraft  and 
between  fiscal  years.  For  example,  the  full  range  of  core  missions  for  an  F-16  Block  50  in 
fiscal  year  201 5  included  three  primary  missions — defensive  counter  air,  offensive  counter 
air-escort,  and  offensive  counter  air-suppression  of  enemy  air  defenses — and  four 
secondary  missions — air  interdiction/offensive  counter  air,  close  air  support,  counter  fast 
attack  craft/fast  inshore  attack  craft,  and  red  air.  In  contrast,  the  core  missions  for  an  F-22 
in  fiscal  year  201 5  included  three  primary  missions — defensive  counter  air,  offensive 
counter  air,  and  offensive  counter  air-attack  operations — and  one  secondary  mission — red 
air. 

23We  interviewed  commanders  from  seven  of  the  thirty-one  fighter  squadrons  that  were 
active  in  fiscal  year  201 5  and  the  commander  from  an  F-35  squadron  that  became 
operational  in  August  2016.  The  views  we  obtained  from  these  interviews  are  not 
generalizable,  but  they  reflect  a  wide  range  of  perspectives  across  the  combat  air  forces. 

24According  to  the  F-35  squadron  commander  we  spoke  with,  maintenance  will  be  fully 
manned  until  fiscal  year  2019. 
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shortfalls  in  maintenance  personnel  limit  their  ability  to  produce  the 
number  of  aircraft  required  to  meet  annual  live-fly  training  requirements. 


Squadron  commanders  we  interviewed  also  identified  factors  that  can 
limit  the  time  aircrews  have  available  to  meet  annual  training 
requirements,  including  the  following: 

High  deployment  rates — Four  squadron  commanders  told  us  that  high 
deployment  rates  limited  the  ability  of  aircrews  to  conduct  annual 
training  across  the  full  range  of  core  missions  between  deployments. 
Certain  fighter  squadrons  have  been  deploying  every  eighteen 
months  for  a  6-month  period  each  time.  During  these  6-month 
deployments,  squadrons  have  been  flying  primarily  close  air  support 
missions  and  are  unable  to  be  proficient  across  their  full  range  of  core 
missions.  When  they  return  from  deployment,  squadrons  have  12 
months  to  rebuild  proficiency  across  their  full  range  of  core  missions 
before  their  next  six-month  deployment.  Some  squadron  commanders 
told  us  that  this  high  deployment  rate  typically  reduces  aircrews’  12- 
month  training  schedule  by  at  least  2  months,  because  they  must 
focus  on  close  air  support  training  in  preparation  for  the  next 
deployment. 

Adversary  air  tasking — Five  squadron  commanders  told  us  that  their 
squadrons  have  been  tasked  heavily  to  provide  adversary  air  for  large 
force  exercises,25  such  as  joint  and  multinational  “Red  Flag”  exercises 
designed  to  provide  realistic  training  in  a  contested  or  degraded 
environment.26  According  to  Air  Force  officials,  the  demand  for 
adversary  air  at  large  force  exercises  has  increased  because  in  fiscal 
year  201 5  one  of  the  two  dedicated  adversary  air  squadrons  at  Nellis 
Air  Force  Base  was  eliminated  because  of  budgetary  reductions.  To 
help  mitigate  this  reduction  in  adversary  air  capability,  Air  Combat 
Command  increased  the  tasking  of  fighter  squadrons  to  provide  this 
capability.  For  example,  an  F-15E  squadron  commander  told  us  that 
his  squadron  will  have  been  tasked  3  of  the  first  9  months  of  fiscal 


25Adversary  air  or  “red  air”  missions  are  those  in  which  the  aircrews  play  the  role  of  an 
adversary  threat  in  support  of  aircrews  flying  a  “blue”  (U.S.  and  allied  force)  training  sortie. 

26According  to  the  Air  Force,  Red  Flag  is  the  U.S.  Air  Force’s  premier  air-to-air  combat 
training  exercise.  Participants  often  include  both  United  States’  and  allied  nations’  combat 
airforces.  The  exercise  provides  aircrews  the  experience  of  multiple,  intensive  air  combat 
sorties  in  the  safety  of  a  training  environment. 
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year  201 6  to  provide  adversary  air  for  various  exercises,  including  a 
Red  Flag  exercise.  According  to  one  squadron  commander, 
participation  as  adversary  air  in  these  large  force  exercises  can  take 
up  to  2  months  out  of  the  squadron’s  training  cycle  to  prepare  for  and 
execute  this  role.  This  limits  the  time  available  to  conduct  annual 
training  for  the  squadron’s  primary  and  secondary  missions. 

Squadron  manpower  shortfalls — Five  squadron  commanders  told  us 
that  shortfalls  in  their  squadron  manning  can  limit  the  time  aircrews 
have  to  meet  their  annual  training  requirements.  Specifically,  with 
fewer  aircrews  assigned  to  a  squadron,  aircrews  are  required  to  take 
on  a  larger  share  of  the  administrative  functions  of  the  squadron, 
including  scheduling  training  and  weapons  evaluation.  This  challenge 
can  be  greater  for  squadrons  operating  under  the  total  force 
integration  construct,  which  uses  aircrews  from  reserve  component 
squadrons  to  supplement  the  aircrews  in  active-duty  squadrons.  For 
example,  the  two  F-22  squadrons  in  the  1st  Fighter  Wing  are 
integrated  with  F-22  aircrews  from  the  192nd  Fighter  Wing  of  the 
Virginia  Air  National  Guard.  As  a  result,  there  are  fewer  active-duty 
aircrews  in  the  squadron.  However,  according  to  both  F-22  squadron 
commanders,  these  aircrews  are  required  to  perform  the 
administrative  duties  for  the  full  squadron,  because  the  Air  National 
Guard  aircrews  are  typically  available  only  on  the  weekends,  and  their 
time  with  the  squadron  is  used  for  training.  Both  F-22  squadron 
commanders  we  spoke  with  told  us  that  even  if  maintenance 
personnel  could  generate  the  number  of  aircraft  needed  to  meet  their 
annual  training  requirements,  the  squadrons  would  be  unable  to 
conduct  additional  training,  because  their  aircrews’  availability  is 
limited  by  their  other  assigned  responsibilities  within  the  squadrons. 

In  addition  to  factors  that  limit  the  ability  of  squadrons  to  meet  their  annual 
training  requirements,  the  squadron  commanders  we  spoke  with  also 
cited  a  number  of  factors  that  affected  the  training  that  aircrews  are  able 
to  accomplish.  The  extent  to  which  these  factors  affected  training  differed 
between  fourth-  and  fifth-generation  aircraft.  For  fourth-generation 
aircraft,  five  squadron  commanders  told  us  that  the  considerable  number 
of  taskings  to  perform  adversary  air  at  large  force  exercises  do  not 
provide  effective  training  opportunities.  Additionally,  some  of  these 
commanders  expressed  concern  that  adversary  air  taskings  would 
increase  and  that  recent  changes  to  their  annual  training  requirements 
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may  mask  this  problem.  Specifically,  prior  to  fiscal  year  2016,  the  annual 
training  requirements  for  certain  combat  aircrews  set  a  cap  on  the  annual 
number  of  adversary  air  sorties  that  could  count  toward  their  combat 
mission  ready  status.27  However,  in  the  fiscal  year  2016  training 
requirements,  that  cap  was  removed,  allowing  for  all  adversary  air  sorties 
to  count  toward  the  designation  of  combat  mission  ready.  Two  squadron 
commanders  told  us  they  were  concerned  that  this  change  could  result  in 
masking  the  readiness  ratings  for  squadrons,  since  overflying  adversary 
aircraft  sortie  requirements  provides  limited  training  compared  with  their 
full  range  of  core  missions,  but  still  allows  the  aircrews  to  be  designated 
as  combat  mission  ready.  According  to  Air  Combat  Command  officials, 
this  change  was  made  to  encourage  aircrews  to  accurately  log  adversary 
air  sorties.  These  officials  stated  that  previously  some  aircrews  may  have 
been  flying  in  an  adversary  air  role,  but  would  complete  some  other 
limited  type  of  training  maneuver  so  that  they  could  record  the  sortie  as  a 
different  mission  that  would  count  toward  their  combat  mission  ready 
status. 

For  fifth-generation  aircraft,  all  three  squadron  commanders  we 
interviewed  cited  various  factors  affecting  training,  including  airspace  and 
training  range  limitations  and  the  availability  of  adversary  air  to  fly  against. 
For  example,  two  F-22  squadron  commanders  at  Langley  Air  Force  Base 
told  us  that  the  size  of  the  base’s  airspace  limits  their  ability  to  train  for 
their  more  complex  missions,  including  offensive  counter  air  and 
defensive  counter  air  missions,  because  the  available  airspace  constrains 
the  capabilities  they  are  able  to  deploy.  Additionally,  all  three  of  the 
commanders  for  squadrons  flying  fifth-generation  aircraft  told  us  that 
limits  in  training  range  capabilities,  such  as  threat  replicators  and  targets, 
affected  the  training  completed  at  smaller  regional  training  ranges,  as  well 
as  at  larger  training  ranges  such  as  the  Utah  Test  and  Training  Range 
and  the  Nevada  Test  and  Training  Range.  According  to  these  officials, 
the  training  ranges  lack  many  of  the  more  advanced  threat  replication 
systems  that  can  challenge  F-35  and  F-22  capabilities  and  provide 
effective  training  across  their  full  range  of  core  missions.  According  to 
training  range  officials  at  the  Nevada  Test  and  Training  Range,  the 


27ln  the  annual  training  requirements  for  fiscal  year  2015,  both  experienced  and 
inexperienced  F-16  aircrew  could  count  only  14  adversary  air  sorties  toward  their  combat 
mission  ready  status,  while  F-15E  aircrews  could  count  only  8  for  experienced  aircrews 
and  9  for  inexperienced  aircrews. 
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current  range  capabilities  are  adequate  to  replicate  only  a  “first  Gulf  War”- 
era  threat  scenario. 

Air  Combat  Command  has  implemented  some  initiatives  to  help  increase 
the  number  of  training  sorties  that  aircrews  are  able  to  conduct.  However, 
these  initiatives  can  further  affect  the  training  that  is  completed.  For 
example,  Air  Combat  Command  has  received  increased  funding  for  “Red 
Flag”  exercises  to  enable  a  fourth  annual  exercise  at  Nellis  Air  Force 
Base.  These  exercises  can  provide  a  high-end,  realistic  training 
environment  for  aircrews.  However,  officials  explained  that  Red  Flag 
exercises  require  a  significant  adversary  air  presence  to  provide  high 
quality  training  for  the  participants.  As  discussed  above,  these  exercises 
can  result  in  squadrons  being  overtasked  to  fly  as  the  adversary  force, 
which  can  further  limit  aircrews’  availability  to  train  to  their  full  range  of 
core  missions.  To  help  mitigate  the  burden  on  squadrons,  in  December 
2015,  the  Air  Force  began  contracting  with  a  private  company  to  provide 
adversary  air  capabilities  on  a  short-term  basis  to  support  exercises  at 
Nellis  Air  Force  Base.  According  to  Air  Force  officials,  in  fiscal  year  2017, 
the  Air  Force  plans  to  spend  about  $25  million  to  contract  for  six  A-4 
aircraft  (a  legacy  fighter  platform  first  developed  in  the  1950s)  to  provide 
adversary  air  for  two  months  of  exercises. 

In  addition,  squadron  commanders,  in  coordination  with  maintenance 
groups,  have  developed  some  initiatives  to  increase  the  number  of  sorties 
they  are  able  to  produce,  but  these  initiatives  can  further  limit  the  training 
that  is  completed.  For  example,  seven  squadron  commanders  told  us 
they  have  implemented  the  use  of  “hot  pits” — a  practice  where  the  aircrew 
flies  a  sortie,  lands  and  refuels,  and  then  flies  another  sortie  without 
shutting  down  the  aircraft.  This  practice  minimizes  the  aircraft  repairs  that 
may  arise  between  sorties  if  an  aircraft  is  shut  down  and  then  restarted. 
Five  squadron  commanders  told  us  that  they  also  occasionally  use  tanker 
aircraft  to  refuel  their  squadrons  in  the  air  during  training.  Both  “hot 
pitting”  and  aerial  refueling  practices  produce  an  additional  sortie  for  the 
aircrews  that  day.  However,  squadron  commanders  told  us  that  such 
practices  prevent  the  aircrews  from  getting  the  full  benefit  from  both 
training  sorties.  Specifically,  these  practices  do  not  allow  the  aircrews  to 
receive  a  mission  brief  and  debrief  for  both  sorties,  a  critical  learning 
component  of  effective  training.  In  addition,  squadron  commanders  told 
us  that  without  the  benefit  of  a  full  briefing  for  the  second  training  sortie, 
the  aircrews  are  not  able  to  conduct  their  more  complex  missions  that 
require  additional  instruction  to  be  effective.  Figure  5  shows  an  F-22 
aircraft  conducting  a  “hot  pit”  refueling  and  an  F-15C  aircraft  conducting 
an  aerial  refueling. 
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Figure  3:  F-22  Aircraft  Conducting  a  “Hot  Pit”  Refueling  and  an  F-15C  Aircraft  Conducting  an  Aerial  Refueling 


Source:  Defense  Video  and  Imagery  Distribution  System.  |  GAO-16-864 


The  Air  Force  has  a  number  of  efforts  under  way  to  study  or  address 
some  of  the  factors  that  limit  the  ability  of  fighter  squadrons  to  meet 
annual  training  requirements  and  that  affect  the  training  that  aircrews  are 
able  to  accomplish,  but  it  will  likely  take  time  before  the  full  benefit  of 
these  efforts  can  be  assessed.  For  example,  the  Air  Force  has 
announced  several  initiatives  to  manage  the  shortage  of  maintenance 
personnel,  including  hiring  additional  maintenance  personnel,  temporarily 
transitioning  active-duty  maintenance  units  from  some  legacy  aircraft,  and 
offering  selective  reenlistment  bonuses  as  an  incentive  to  improve 
retention,  among  other  initiatives.  According  to  Air  Force  officials,  it  may 
take  several  years  before  newly  hired  maintenance  personnel  will  have 
the  training  and  experience  they  need  in  order  to  improve  aircraft 
availability  rates.  In  addition,  Air  Combat  Command  officials  told  us  that 
there  is  an  ongoing  study  reviewing  adversary  air  requirements.  Further, 
the  Air  Force  has  taken  steps  to  upgrade  its  training  infrastructure, 
including  training  ranges.  For  example,  in  its  fiscal  year  2017  budget 
request,  the  Air  Force  requested  a  funding  increase  of  about  $35  million 
for  the  maintenance  and  sustainment  of  21  training  ranges,  including 
installation  of  additional  threat  emitters  and  expansion  of  communication 
capabilities  between  air  and  ground  forces. 
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The  Air  Force  Does  Not 
Systematically  Evaluate 
the  Effectiveness  of 
Completed  Training 
against  Expectations 


Air  Force  processes  used  to  record  and  monitor  combat  fighter 
squadrons’  annual  training  are  focused  on  the  frequency  of  training 
completed  to  meet  annual  requirements  and  do  not  include  a  systematic 
evaluation  of  the  effectiveness  of  that  training  against  established 
expectations.  Squadron  commanders  track  and  report  monthly  and  3- 
month  “look-backs”  to  determine  if  aircrews  are  meeting  their  prescribed 
annual  training  requirements.  These  assessments  emphasize  the 
frequency  of  training  and  inform  the  designation  of  aircrews  as  combat 
mission  ready.28  According  to  Air  Force  officials,  squadron  commanders 
are  delegated  with  the  authority  to  determine  whether  a  sortie  should 
“count”  toward  meeting  the  annual  training  requirements.  These 
determinations  are  based  on  Air  Force  guidance — including  AFI  11-2, 
Volume  1 — that  establish  common  training  standards  for  each  type  of 
combat  aircraft  and  documentation  supporting  the  Ready  Aircrew 
Program  that  provides  a  list  of  training  expectations  that  include  the  types 
of  events  that  should  be  included  in  a  training  mission  to  be  considered 
effective.29  For  example,  the  Ready  Aircrew  Program  tasking 
memorandums  establish  when  a  non-effective  sortie  should  be  recorded, 
for  example  when  training  limitations  are  caused  by  weather  conditions, 
aircraft  maintenance  or  weapon  malfunctions,  or  cancellation  of  training 
support  elements — which  can  include  adversary  air  or  air-refueling 
assets. 


Air  Combat  Command  uses  quantitative  data  on  the  completion  of  annual 
training  requirements  and  qualitative  information  on  broad  factors  that 
affect  training  in  various  forums  that  discuss  training.  However,  our  review 
of  documentation  supporting  these  forums  found  that  the  Air  Force  does 
not  systematically  evaluate  the  extent  to  which  completed  training  met 
expectations  or  the  degree  to  which  certain  challenges  have  affected 
training.  In  our  review,  we  identified  two  forums  that  Air  Combat 


28Aircrews  that  do  not  meet  the  training  requirements  during  the  look-back  period  are 
designated  as  non-combat  mission  ready  or  non-basic  mission  capable,  or  the  aircrew 
may  be  placed  in  probation  status  for  one  month  at  the  squadron  commander’s  discretion. 
Aircrews  designated  as  non-combat  mission  ready  or  non-basic  mission  capable  must 
complete  a  recertification  program  established  by  the  squadron  commander  to  regain 
combat  mission  ready  or  basic  mission  capable  status,  while  aircrews  on  probation  must 
reestablish  a  complete  one-month  look-back  by  the  end  of  the  probation  period. 

29For  example,  for  an  F-22  aircrew  conducting  an  offensive  counter  air  mission,  the 
training  expectations  include  events  such  as  multi-ship  formations,  high/medium  altitude 
operations,  and  rules  of  engagement  compliance,  among  others. 
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Command  uses  to  oversee  training  completion  and  discuss  training 
challenges.  These  forums  are  focused  on  a  review  of  training  completion 
data  and  include  discussions  of  factors  that  affect  training  completion  and 
the  effectiveness  of  training.  For  example,  at  the  end  of  each  fiscal  year, 
Air  Combat  Command  aggregates  combat  fighter  squadrons’  training 
completion  data  for  its  end-of-year  training  and  readiness  review.  Tables 
are  compiled  by  squadron  and  by  aircraft  type  to  provide  a  comparison  of 
annual  training  completion  data  against  each  aircraft’s  full  range  of  core 
missions.  According  to  Air  Combat  Command  officials,  the  squadron 
commanders  are  required  to  submit  a  list  of  qualitative  factors  that,  in 
general,  affected  the  completion  and  effectiveness  of  the  squadron’s 
training.  These  factors  are  then  discussed  at  an  end-of-year  training  and 
readiness  review,  which  is  an  informational  briefing  used  to  inform  the 
state  of  readiness  of  the  combat  air  forces. 

In  addition,  the  annual  Realistic  Training  Review  Board  provides  another 
forum  for  discussion  between  Air  Combat  Command  and  leadership  at 
the  group  commander  level  on  training  challenges.  According  to  officials 
and  our  review  of  documentation  discussed  during  these  forums,  these 
discussions  are  focused  on  a  number  of  issues  that  affect  training,  such 
as  funding  for  the  flying-hour  program  and  distributed  mission  operations. 
Further,  the  Board  identifies  a  number  of  action  items  related  to  training 
that  need  to  be  addressed.  For  example,  output  from  the  2015  Board 
included  actions  directed  toward  improving  cruise  missile  threat 
replication,  targets  for  F-22  ranges,  and  opportunities  for  certain  weapons 
employment  for  F-15C  squadrons.  While  these  forums  provide  an 
opportunity  for  Air  Force  officials  to  discuss  and  identify  training 
challenges,  neither  includes  a  systematic  evaluation  of  the  effectiveness 
of  training  against  established  expectations  in  order  to  identify  limitations 
to  the  effectiveness  of  training. 

The  lack  of  a  systematic  evaluation  of  the  effectiveness  of  training  against 
established  expectations  has  affected  the  Air  Force’s  ability  to  make 
informed  resource  decisions  to  improve  its  training  infrastructure.  For 
example,  training  range  officials  from  both  the  Nevada  and  Utah  Test  and 
Training  Ranges  told  us  that  without  information  on  the  gaps  between 
aircraft  training  requirements  and  current  range  capabilities,  they  were  not 
well  positioned  to  make  informed  decisions  on  where  to  focus  their 
resources  to  improve  range  capabilities.  An  official  from  the  Nevada  Test 
and  Training  Range  told  us  that  identifying  training  gaps  is  becoming 
even  more  important,  because  the  new  F-35  aircraft  will  have  additional 
range  requirements  that  exceed  those  of  fourth-generation  fighter  aircraft. 
However,  under  current  Air  Force  guidance,  the  Nevada  Test  and 
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Training  Range  is  required  to  provide  capabilities  only  for  large  force 
exercises  and  does  not  currently  possess  the  range  capabilities  needed 
to  support  a  number  of  training  events  identified  in  the  training 
expectations  for  the  F-35,  including  close  air  support  in  an  urban  terrain 
and  certain  attack  events,  among  others. 

We  found  that  the  Air  Force  does  not  systematically  evaluate  the 
effectiveness  of  training  against  established  expectations  for  two  reasons. 
First,  Air  Combat  Command  has  not  established  the  desired  learning 
objectives  and  training  support  elements  needed  to  accomplish  the 
training  expectations  in  its  annual  Ready  Aircrew  Program  tasking 
memorandums.  Our  prior  work  on  strategic  training  notes  that  agencies 
should  systematically  evaluate  the  effectiveness  of  their  training  efforts 
and  collect  data  corresponding  to  established  training  objectives  to 
continually  improve,  deliver,  and  enhance  training  and  ensure  the 
effective  allocation  of  resources.30  In  contrast  to  the  Ready  Aircrew 
Program  tasking  memorandums,  we  found  that  documentation 
underpinning  other  types  of  aircrew  training,  such  as  mission  qualification 
training  and  upgrade  training,  provide  more  detailed  information  on  critical 
features  needed  to  measure  the  effectiveness  of  training  sorties.  For 
example,  the  upgrade  training  syllabus  for  the  F-22  provides  desired 
learning  objectives  for  an  aircrew  to  conduct  an  effective  defensive 
counter  air  sortie  and  information  on  the  number  of  adversary  aircraft 
required  and  the  period  of  time  that  should  be  spent  defending  a  target. 
However,  the  Ready  Aircrew  Program  tasking  memorandum  for  the  F-22 
does  not  include  similar  information  on  an  effective  defensive  counter  air 
sortie.  Instead,  it  provides  only  a  list  of  expectations  that  could  be 
included  in  a  training  mission,  along  with  information  on  the  desired 
amounts  of  adversary  air  needs. 

Second,  the  current  process  used  by  squadrons  to  report  information  on 
their  training  completion  does  not  collect  data  on  the  effectiveness  of  the 
training  against  established  expectations.  Documentation  supporting  the 
Ready  Aircrew  Program  notes  that  credible  and  detailed  training 
accomplishment  reporting  provides  an  accurate  assessment  of  readiness 
and  the  data  needed  to  advocate  for  adequate  resources.  This 
documentation  notes  that  Air  Combat  Command  expects  to  see  data  that 


30GAO-04-546G. 
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document  the  underlying  challenges  to  providing  an  effective  training 
program.  However,  the  form  that  aircrews  are  required  to  complete 
following  a  training  sortie  requires  them  to  record  the  number  of  flying 
hours  and  other  information,  such  as  the  type  of  mission,  but  does  not 
capture  information  on  the  ability  of  aircrews  to  meet  the  training 
expectations  established  in  the  Ready  Aircrew  Program.  For  example,  the 
form  does  not  provide  the  opportunity  to  record  the  training  support 
elements  involved  in  the  sortie,  such  as  the  number  of  adversary  aircraft 
involved  and  the  threat  replicators  available.  In  the  absence  of  a  process 
to  obtain  this  information,  squadron  commanders  are  also  unable  to 
collect  data  corresponding  to  the  effectiveness  of  annual  training 
completed  to  meet  expectations  or  the  degree  to  which  certain  challenges 
affected  training.  One  squadron  commander  we  interviewed  has 
developed  internal  guidance  for  aircrews  to  determine  whether  a  sortie 
was  effective  and  should  count  toward  the  annual  training  requirements. 
This  commander  provides  an  annual  guidance  letter  to  be  used  within  the 
squadron  that  sets  the  minimum  requirements  that  must  be  met  to  log  an 
effective  training  sortie.  However,  based  on  our  interviews  with  other 
squadron  commanders,  this  practice  is  not  being  consistently  performed 
across  all  squadrons. 

Officials  from  Air  Combat  Command  told  us  that  they  had  not  developed 
more  detailed  expectations  for  combat  fighter  squadron  commanders  to 
measure  and  collect  data  on  the  effectiveness  of  annual  training  for 
several  reasons.  These  officials  stated  that  Air  Force  instructions  and  the 
Ready  Aircrew  Program  provide  squadron  commanders  with  the  authority 
to  determine  if  a  training  sortie  was  effective  and  should  be  counted 
toward  meeting  annual  training  requirements.  Further,  these  officials 
noted  that  aircrews  have  been  limited  in  their  ability  to  complete  annual 
training  requirements  due  to  the  factors  previously  discussed  in  this  report 
and  that  having  more  detailed  expectations  for  measuring  the 
effectiveness  of  annual  training  could  make  it  more  difficult  for  aircrews  to 
achieve  a  combat  mission  ready  status.  In  addition,  Air  Combat 
Command  officials  stated  that  the  need  to  complete  training  against  more 
detailed  expectations  and  collect  information  on  training  effectiveness 
should  be  balanced  against  the  time  and  other  factors  that  would  be 
required  to  collect  the  information. 

Given  the  range  of  factors  that  have  limited  the  ability  of  combat  fighter 
squadrons  to  complete  annual  training  requirements,  having  a  consistent 
basis  for  monitoring  training  results  is  critical  in  tracking  progress  toward 
achieving  the  Air  Force’s  goal  of  training  units  for  the  full  range  of  core 
missions.  Moreover,  Air  Combat  Command  has  stated  its  goal  of  moving 
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to  a  training  model  based  on  the  effectiveness  of  training,  using 
quantifiable  levels  of  performance,  rather  than  the  current  model  focused 
on  the  frequency  of  training.  Without  a  process  to  systematically  measure 
and  collect  data  on  the  effectiveness  of  training  against  established 
expectations,  the  Air  Force  will  not  be  well  positioned  to  improve,  deliver, 
and  enhance  training  or  allocate  resources  to  address  factors  that  limit 
the  effectiveness  of  training. 


Air  Force  Planning 
Documents  for  Virtual 
Training  Do  Not 
Include  All  Desirable 
Characteristics  of  a 
Comprehensive 
Strategy 


The  Air  Force’s  virtual  training  plans  do  not  include  all  desirable 
characteristics  of  a  comprehensive  strategy,  such  as  a  risk-based 
investment  strategy  or  a  time  line  for  addressing  training  needs.  We  have 
previously  reported  that  when  addressing  national  or  department-wide 
issues,  it  is  standard  practice  for  organizations  to  have  a  strategy  that 
includes  desirable  characteristics  such  as  establishing  goals  and 
objectives,  identifying  actions  for  addressing  those  objectives,  allocating 
resources,  identifying  roles  and  responsibilities,  and  measuring 
performance  against  objectives,  among  others.31 

In  recent  years,  the  Air  Force  has  developed  some  planning  documents 
to  guide  its  virtual  training  efforts.  In  2013,  for  example,  the  Headquarters 
Air  Force  issued  its  United  States  Air  Force  Live  Virtual  Constructive 
Operational  Training  Flight  Plan.  The  flight  plan  was  the  first  Air  Force 
level  strategic  document  to  address  LVC  operational  training  from  an  Air 
Force-wide  perspective.  The  document  established  the  Air  Force’s  broad 
vision  for  the  future  of  the  LVC  operational  training  environment  and 
highlighted  areas  that  needed  attention  to  advance  the  program  and 
realize  its  full  potential.  In  addition,  in  2014,  Air  Combat  Command  issued 
the  Combat  Air  Force  Live,  Virtual,  Constructive  Vision,  which  describes 
the  command’s  goals  for  the  integration  of  LVC  operational  training 
capabilities,  along  with  the  priority  tasks  and  general  timeframes  for 
implementation.  Further,  in  2015,  Air  Combat  Command  released  the 
Combat  Air  Force  Live,  Virtual,  Constructive  Roadmap  that  sets  near-, 
mid-,  and  far-term  tasks  that  are  required  to  better  define  needed  training 
capabilities. 


31GAO-04-408T. 
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While  the  Air  Force  has  issued  these  planning  documents  to  help  guide 
its  LVC  operational  training  initiatives  for  the  combat  air  forces,  they  do 
not  include  all  of  the  characteristics  we  have  identified  as  being  desirable 
in  comprehensive  strategies.32  Based  on  our  analysis  of  the  United  States 
Air  Force  Live  Virtual  Constructive  Operational  Training  Flight  Plan  and 
the  Combat  Air  Force  Live,  Virtual,  Constructive  Vision,  we  found  that  the 
Air  Force  has  only  partially  addressed  or  has  not  addressed  most  of  the 
desirable  characteristics,  as  shown  in  table  I.33 


Table  1:  Extent  to  Which  the  United  States  Air  Force  Live  Virtual  Constructive  Operational  Training  Flight  Plan  and  Combat 
Air  Force  Live,  Virtual,  Constructive  Vision  Address  GAO’s  Desirable  Characteristics  for  Comprehensive  Strategies 


Desirable  Characteristic 

Description 

United  States  Air  Force  Live 

Virtual  Constructive  Operational 
Training  Flight  Plan 

Combat  Air  Force  Live,  Virtual, 
Constructive  Vision 

Problem  definition  and  risk 
assessment 

Defines  the  particular 
problem  the  strategy  is 
directed  toward  and 
includes  an  analysis  of 
threats  to  critical 
operations. 

Partially  Addresses. 

Discusses  risk  and  defines  the 
program  but  does  not  include  a  risk 
assessment  that  identifies  the  threat 
and  vulnerabilities  the  strategy  is 
directed  toward. 

Partially  Addresses. 

Discusses  the  problem  definition  but 
does  not  include  a  risk  assessment 
that  identifies  the  threat  and 
vulnerabilities  the  strategy  is  directed 
toward. 

Purpose,  scope,  and 
methodology 

Discusses  why  the  strategy 
was  produced,  the  scope  of 
its  coverage,  and  the 
process  by  which  it  was 
developed. 

Partially  Addresses. 

Discusses  the  strategy’s  purpose, 
but  does  not  fully  discuss  its  scope 
and  methodology,  including  the 
major  functions  and  activities  it 
covers  and  the  process  of  how  it  was 
developed. 

Partially  Addresses. 

Discusses  why  the  strategy  was 
produced  and  its  scope,  but  does  not 
describe  the  process  of  how  it  was 
developed. 

32We  assessed  these  two  documents  against  criteria  that  we  developed  for  national 
strategies.  These  criteria  are  cited  in  GAO,  Combating  Terrorism:  Evaluation  of  Selected 
Characteristics  in  National  Strategies  Related  to  Terrorism,  GAO-04-408T  (Washington, 
D.C.:  Feb.  3,  2004).  Given  the  size  and  complexity  of  the  Air  Force’s  virtual  training 
efforts,  we  concluded  that  a  comprehensive  strategy  to  guide  these  efforts  would  be 
similar  in  scope  and  would  need  to  include  characteristics  similar  to  those  of  a  national 
strategy. 

33For  the  purposes  of  our  review,  we  focused  our  evaluation  on  planning  efforts  at 
Headquarters  Air  Force,  which  is  responsible  for  Air  Force-wide  virtual  training  and  at  Air 
Combat  Command,  which  is  responsible  for  virtual  training  within  the  combat  air  forces. 
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Desirable  Characteristic 

Description 

United  States  Air  Force  Live 

Virtual  Constructive  Operational 
Training  Flight  Plan 

Combat  Air  Force  Live,  Virtual, 
Constructive  Vision 

Organizational  roles, 
responsibilities,  and 
coordination 

Identifies  who  will  be 
implementing  the  strategy, 
defines  what  roles  will  be 
compared  to  others,  and 
discusses  mechanisms  for 
coordination  of  efforts. 

Addresses. 

Addresses. 

Integration  and 
implementation 

Discusses  how  the  strategy 
relates  to  other  strategies’ 
goals,  objectives,  and 
activities. 

Partially  Addresses. 

Discusses  supporting  DOD  guidance 
and  strategies,  but  does  not  provide 
details  on  implementation  plans. 

Partially  Addresses. 

Describes  broad  phases  of 
implementation,  but  does  not  provide 
guidance  on  how  implementing 
parties  link  their  roles, 
responsibilities,  and  capabilities. 

Goals,  subordinate 
objectives,  activities, 
measures  of  performance, 
and  monitoring  of  progress 

Defines  what  the  strategy 
is  trying  to  achieve  and 
identifies  steps  to  achieve 
those  results,  as  well  as 
the  priorities,  milestones, 
and  measures  to  gauge 
results  and  monitor 
progress. 

Partially  Addresses. 

Includes  end-state  goals,  along  with 
some  objectives,  activities,  and 
milestones,  but  no  specific 
performance  measures  or  process  to 
monitor  progress  are  identified. 

Partially  Addresses. 

Includes  objectives  across  three 
implementation  phases  but  no 
specific  performance  measures  or 
process  for  monitoring  progress  are 
identified. 

Resources,  investments, 
and  risk  management 

Identifies  what  the  strategy 
will  cost,  the  sources  and 
types  of  resources,  and 
where  resources  and 
investments  should  be 
targeted. 

Does  not  address. 

Does  not  address. 

Source:  GAO  analysis  of  Air  Force  virtual  training  strategic  documents  I  GAO-16-864. 

Note:  Explanations  are  provided  only  for  why  an  element  partially  meets  GAO  criteria  for  desirable 
characteristics  of  strategy  documents.  A  document  “addresses”  a  characteristic  when  it  explicitly  cites 
all  elements  of  the  characteristic,  even  if  it  lacks  specificity  and  details  and  thus  could  be  improved 
upon.  A  document  “partially  addresses”  a  characteristic  when  it  explicitly  cites  some,  but  not  all 
elements  of  the  characteristic.  Within  our  designation  of  “partially  addresses”  there  is  wide  variation 
between  a  document  that  addresses  most  of  the  elements  of  a  characteristic  and  a  document  that 
addresses  few  of  those  elements.  A  document  “does  not  address”  a  characteristic  when  it  does  not 
explicitly  cite  or  discuss  any  elements  of  a  characteristic,  or  when  any  implicit  references  are  either 
too  vague  or  general. 


For  one  characteristic,  we  found  that  both  of  the  planning  documents 
contained  the  desirable  elements.  For  example,  United  States  Air  Force 
Live  Virtual  Constructive  Operational  Training  Flight  Plan  establishes  the 
roles  and  responsibilities  for  organizations  involved  in  the  implementation 
of  LVC  operational  training  initiatives,  including  Headquarters  Air  Force 
and  the  major  commands,  such  as  Air  Combat  Command.  It  also 
discusses  the  governance  structure  and  the  role  of  primary  support 
organizations,  such  as  the  LVC  Council  of  Colonels  and  the  Distributed 
Warfare  Integration  Cell,  which  are  intended  to  identify  and  prioritize  LVC 
operational  training  gaps  and  shortfalls  that  cut  across  the  major 


Page  29 


GAO-16-864  Air  Force  Training 


commands.  For  other  characteristics,  we  found  that  some  of  the  elements 
were  included,  but  not  all.  For  example,  the  Combat  Air  Force  Live, 

Virtual,  Constructive  Vision  identifies  a  number  of  implementation  tasks 
across  a  range  of  implementation  phases,  but  it  does  not  establish 
priorities,  specific  milestones,  or  performance  measures.  Our  prior  work 
has  shown  that  these  elements  can  help  to  ensure  effective 
implementation  and  accountability  to  achieve  results.  Further,  for  some 
characteristics,  we  determined  that  neither  planning  document  contained 
any  of  the  desirable  elements.  For  example,  neither  of  the  two  documents 
identified  what  the  virtual  training  efforts  will  cost,  the  sources  and  types 
of  resources  required,  or  the  prioritization  of  needed  investments.  Our 
prior  work  has  shown  that  a  risk-based  investment  strategy  may  be  used 
to  define  and  prioritize  related  resource  and  operational  requirements,  as 
well  as  to  develop  a  time  line  for  obtaining  capability  needs.  A  risk-based 
investment  strategy  includes  five  key  phases:  (1)  setting  strategic  goals 
and  objectives  and  determining  constraints,  (2)  assessing  risks,  (3) 
evaluating  alternatives  for  addressing  these  risks,  (4)  selecting  the 
appropriate  alternatives,  and  (5)  implementing  the  alternatives  and 
monitoring  the  progress  made  and  results  achieved.34 

We  identified  several  areas  where  the  Air  Force  is  making  significant 
investments  in  virtual  training,  despite  the  lack  of  a  comprehensive 
strategy  needed  to  guide  and  prioritize  these  efforts.  In  some  cases,  this 
has  led  to  investments  that  may  not  maximize  available  Air  Force 
resources.  For  example,  the  Air  Force  is  building  an  F-22  simulator  facility 
at  Nellis  Air  Force  Base.  The  facility  is  expected  to  be  completed  in  fiscal 
year  2017,  at  an  estimated  cost  of  $14  million.  However,  according  to 
officials  at  Nellis  Air  Force  Base,  the  Air  Force  does  not  expect  the  four 
simulators  to  arrive  there  until  fiscal  year  2019  at  the  earliest,  leaving  the 
facility  potentially  unutilized  during  that  interim  period.  Additionally, 
officials  identified  a  number  of  examples  where  the  Air  Force  has  not 
made  cost-efficient  investments  in  virtual  training  systems.  For  instance, 
officials  from  Headquarters  Air  Force  noted  that  different  versions  of  the 
F-16  simulator  have  been  procured  within  the  active,  guard,  and  reserve 
components,  which  create  inefficiencies  among  the  components  as  they 
develop,  procure,  and  sustain  these  differing  simulators. 


34See  GAO,  Intelligence,  Surveillance,  and  Reconnaissance:  DOD  Needs  a  Strategic, 
Risk-Based  Approach  to  Enhance  Its  Maritime  Domain  Awareness,  GAO-1 1-621 
(Washington,  D.C.:  June  20,  2011)  and  GAO-04-408T. 
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Air  Force  officials  also  provided  examples  of  prior  investments  in 
simulators  that  do  not  meet  the  training  needs  of  aircrews  across  their  full 
range  of  core  missions.  For  example,  the  Air  Force  Audit  Agency  reported 
in  2013  that  Air  Force  officials  had  not  adequately  defined  F-22  simulator 
requirements,  because  the  procurement  contracts  did  not  include 
simulator  performance  specifications.  Instead,  all  F-22  simulators  shared 
a  single  training  system  specification  that  provided  insufficient  technical 
detail  as  to  what  the  Air  Force  considered  an  acceptable  product.35  As  a 
result,  F-22  squadrons  at  Langley  Air  Force  Base  have  traveled  to  the  Air 
Combat  Simulator  facility  in  Marietta,  Georgia  to  utilize  the  contractor- 
owned  F-22  simulators  at  that  location.  Officials  from  the  F-22  squadrons 
told  us  that  the  contractor-owned  simulators  provide  higher-quality 
training  than  the  simulators  at  their  home  station  and  allow  them  to  train 
in  a  more  realistic  threat  environment.  For  2015,  the  cost  of  using  the  Air 
Combat  Simulator  facility  in  Marietta  was  around  $10,000  per  day,  plus 
travel  costs  for  the  aircrews.  According  to  an  official  from  Air  Combat 
Command,  F-22  squadrons  at  Langley  Air  Force  Base  utilized  the  Air 
Combat  Simulator  facility  for  5  days  in  calendar  year  201 5  and,  as  of 
February  2016,  have  utilized  the  facility  for  1 1  days  in  calendar  year 
2016.  In  addition,  based  on  our  review  of  simulator  utilization  data,  the  F- 
22  simulators  at  Langley  Air  Force  Base  remain  underutilized. 

Air  Combat  Command  has  identified  significant  requirements  for  LVC 
operational  training  capabilities  that  will  require  funding  over  the  next 
several  years.  Specifically,  the  Air  Force  estimates  that  LVC  operational 
training  requirements  will  total  approximately  $3.8  billion  in  funding  for 
fiscal  years  2015  through  2019.  This  includes  funding  to  maintain 
simulator  currency  with  the  aircraft,  modernize  simulator  visual  systems, 
secure  virtual  training  networks,  and  hire  additional  simulator  instructors. 
Air  Force  officials  told  us  that  concern  over  simulator  information 
security — particularly  for  fifth-generation  aircraft — will  require  network 
upgrades  to  ensure  that  the  infrastructure  has  adequate  controls  in  place 
to  protect  data.  According  to  these  officials,  network  security  and 
information  assurance  are  currently  the  most  pressing  challenges  in  the 
Air  Force’s  virtual  training  environment. 


35Air  Force  Audit  Agency,  Acquisition  Management  of  Fifth-Generation  Fighter  Aircraft 
Simulators,  F201 3-0007-L30000  (Feb.  26,  2013). 
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While  the  United  States  Air  Force  Live  Virtual  Constructive  Operational 
Training  Flight  Plan  and  the  Combat  Air  Force  Live,  Virtual,  Constructive 
Vision  demonstrate  some  effort  toward  defining  a  broad  strategy  for  the 
Air  Force’s  virtual  training  efforts,  the  Air  Force  lacks  a  comprehensive 
strategy,  because  it  has  not  fully  refined  its  planning  documents  to 
achieve  its  training  vision,  for  example  by  developing  a  risk-based 
investment  strategy  that  identifies  and  prioritizes  capability  needs  and 
includes  a  time  line  for  addressing  them.  The  Flight  Plan  recognizes  the 
need  for  an  investment  strategy  to  prioritize  and  guide  investments  in 
future  LVC  operational  training  capabilities;  however,  officials  told  us  that 
development  of  an  investment  strategy  has  not  begun.  Additionally,  as  we 
reported  in  2012,  the  Air  Force  does  not  have  a  methodology  to  fully 
account  for  and  track  the  costs  of  its  virtual  training  enterprise. 
Headquarters  Air  Force  officials  told  us  that  they  have  made  some 
progress  in  tracking  operation  and  maintenance  costs  for  virtual  training 
but  none  toward  tracking  investment  funding.  As  we  reported  in  2012, 
identifying  virtual  training  costs  is  challenging,  because  funds  that  support 
virtual  training  are  dispersed  across  multiple  program  elements. 
Headquarters  Air  Force  officials  told  us  that  they  have  identified 
approximately  75  program  elements  that  support  the  LVC  operational 
training  program,  but  that  it  is  difficult  to  determine  if  these  elements 
represent  all  required  funding.  Further,  these  officials  told  us  that  these  75 
program  elements  are  not  consistently  monitored,  because  the 
automation  required  to  monitor  and  conduct  any  trending  analysis  on  the 
program  elements  does  not  currently  exist. 

Given  the  importance  of  an  integrated  LVC  operational  training 
environment  to  the  future  of  Air  Force  training,  a  comprehensive  strategy 
would  help  ensure  that  the  Air  Force  develops  the  capabilities  needed  to 
meet  its  virtual  training  needs.  Without  a  comprehensive  approach  to 
virtual  training  that  includes  a  risk-based  investment  strategy  and  a  time 
line  for  prioritizing  and  addressing  capability  needs,  the  Air  Force  may  be 
slow  to  develop  needed  capabilities  and  therefore  could  increase  risk  for 
fifth-generation  fighter  aircraft,  whose  training  will  require  a  robust, 
realistic  virtual  training  environment. 


Conclusions 


The  Air  Force  has  identified  the  need  to  ensure  a  full-spectrum  capable 
force  that  can  successfully  achieve  missions  to  address  a  broad  range  of 
current  and  emerging  threats.  However,  the  Air  Force  has  not  completed 
the  analysis  and  planning  needed  to  ensure  that  aircrew  training 
requirements  fully  align  with  those  goals.  Specifically,  Air  Combat 
Command  has  not  comprehensively  reassessed  the  assumptions 
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underlying  its  annual  training  requirements,  some  of  which  have  been  in 
place  for  40  years.  Without  fully  reassessing  the  assumptions  underlying 
its  requirements,  the  Air  Force  cannot  be  certain  that  its  annual  training 
plans  are  aligned  with  its  stated  goals  to  ensure  a  full-spectrum  capable 
force  that  can  successfully  achieve  missions  across  a  broad  range  of 
current  and  emerging  threats.  In  addition,  Air  Combat  Command  has  not 
identified  desired  learning  objectives  or  training  support  elements 
necessary  to  accomplish  the  training  expectations  in  its  annual  Ready 
Aircrew  Program  tasking  memorandums.  Having  a  consistent  basis  for 
monitoring  training  results  is  critical  for  tracking  progress  in  achieving  the 
Air  Force’s  goal  of  training  units  for  the  full  range  of  core  missions.  Also, 
without  a  process  to  collect  data  to  assess  the  effectiveness  of  the 
training,  the  Air  Force  may  not  be  well  positioned  to  improve,  deliver,  and 
enhance  training  or  allocate  resources  to  address  factors  that  limit  the 
effectiveness  of  training.  Further,  the  Air  Force  lacks  a  comprehensive 
strategy  for  virtual  training,  because  it  has  not  fully  refined  its  planning 
documents  to  achieve  its  training  vision,  for  example  by  developing  a  risk- 
based  investment  strategy  and  a  time  line  for  prioritizing  its  needs. 
Achieving  this  vision  is  critical;  as  new  fifth-generation  aircraft  become 
operational  they  will  require  a  robust  and  realistic  virtual  training 
environment  to  train  to  their  full  capabilities. 


Recommendations  for  ensure  ^at  annual  training  plans  are  aligned  with  the  Air  Force’s 

stated  goals  to  ensure  that  its  forces  can  successfully  achieve  missions 
Executive  Action  across  a  broad  range  of  current  and  emerging  threats,  we  recommend 

that  the  Secretary  of  Defense  direct  the  Secretary  of  the  Air  Force  to 
comprehensively  reassess  the  assumptions  underlying  its  annual  training 
requirements — including,  but  not  limited  to,  the  total  annual  training 
requirements  by  aircraft,  the  criteria  for  designating  aircrews  as 
experienced  or  inexperienced,  and  the  mix  between  live  and  simulator 
training — and  make  any  appropriate  adjustments  in  future  training  plans. 

To  improve  the  Air  Force’s  ability  to  consistently  monitor  training  results 
and  better  position  it  to  allocate  resources  to  address  factors  that  limit  the 
effectiveness  of  training,  we  recommend  that  the  Secretary  of  Defense 
direct  the  Secretary  of  the  Air  Force  to 

•  establish  desired  learning  objectives  and  training  support  elements 
needed  to  accomplish  the  training  expectations  in  its  annual  Ready 
Aircrew  Program  tasking  memorandums  and 
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•  develop  a  process  to  collect  data  to  assess  the  effectiveness  of 
annual  training  against  these  features. 

To  improve  the  Air  Force’s  ability  to  develop  the  capabilities  needed  to 
meet  its  virtual  training  needs,  we  recommend  that  the  Secretary  of 
Defense  direct  the  Secretary  of  the  Air  Force  to  continue  to  refine  its 
planning  for  virtual  training  to  incorporate  the  desirable  characteristics  of 
a  comprehensive  strategy,  including  developing  a  risk-based  investment 
strategy  that  identifies  and  prioritizes  capability  needs  and  includes  a  time 
line  for  addressing  them. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  the  non-public  sensitive  version  of  this  report,36 
DOD  did  not  concur  with  the  first  and  second  recommendations,  and 
concurred  with  the  third  recommendation.  DOD’s  comments  are  reprinted 
in  their  entirety  in  appendix  II. 

In  its  comments,  DOD  stated  it  did  not  concur  with  our  first 
recommendation,  but  DOD  identified  some  actions  it  was  taking  to 
address  the  intent  of  the  recommendation.  Specifically,  DOD  stated  that 
the  Air  Force  has  already  begun  a  comprehensive  reassessment  of  the 
assumptions  underlying  its  annual  training  requirements  and  that  those 
results  will  be  used  to  influence  future  training  plans.  More  specifically, 
DOD  stated  that  the  Air  Force  has  directed  a  study  that  will  reassess  the 
full  spectrum  of  the  Air  Force’s  Ready  Aircrew  Program  training 
requirements  and  provide  a  range  of  readiness  levels  associated  with 
varying  training  levels.  According  to  DOD’s  comments,  the  estimated 
completion  date  of  the  study  is  September  2017.  DOD  also  stated  that  a 
second  study  is  reviewing  the  factors  that  contribute  to  the  Air  Force’s 
aircrew  experience  designation.  For  example,  the  Air  Force  has 
completed  a  review  of  the  F-22  that  resulted  in  substantive  changes  for 
the  F-22  experiencing  criteria.  DOD’s  comments  further  stated  that  the  Air 
Force  has  directed  a  follow-on  study  for  all  fourth-generation  fighter 
aircraft  with  an  estimated  completion  date  for  this  study  of  September 
2017.  As  we  noted  in  our  report,  the  Air  Force’s  initiatives  represent 
positive  steps  to  study  aspects  of  the  combat  aircrew  annual  training 
requirements.  However,  given  that  these  studies  are  in  the  initial  stages 
or  not  yet  underway,  it  is  unclear  whether  the  scope  of  the  studies  is 


36GAO-1 6-635SU. 
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sufficient  to  address  all  of  the  assumptions  that  need  to  be  revisited. 
Further,  it  is  too  early  to  determine  what  adjustments,  if  any,  the  Air  Force 
will  make  in  its  future  training  plans  in  response  to  the  studies’  results,  as 
we  recommended.  Therefore,  we  believe  the  recommendation  remains 
valid. 

DOD  did  not  concur  with  our  second  recommendation,  stating  that  the  Air 
Force’s  Ready  Aircrew  Program  training  differs  significantly  from  other 
syllabus-directed  courses  of  instruction  and  that  desired  learning 
objectives  for  this  training  are  set  at  the  squadron  level  in  accordance 
with  current  Air  Force  guidance.  Specifically,  the  Ready  Aircrew  Program 
tasking  memorandums  direct  squadron  commanders  to  develop  unit 
training  programs  that  focus  on  their  primary  and  secondary  missions, 
and  that  in  order  to  be  effective,  each  mission  must  successfully  complete 
a  sufficient  number  of  events  applicable  to  that  mission  type,  according  to 
the  squadron  commander.  In  addition,  DOD’s  comments  noted  that  Air 
Force  guidance  provides  higher  headquarters  direction  at  the  appropriate 
level  while  providing  local  commanders  with  the  flexibility  to  set  detailed 
learning  objectives  for  individual  sorties  that  allows  them  to  maximize  the 
training  value  and  efficiency  of  each  sortie.  DOD  also  stated  that  a  table 
in  the  Ready  Aircrew  Program  tasking  memorandums  identifies  the 
desired  adversary  air  ratios,  which  serves  as  the  training  support 
elements  needed  to  accomplish  training  expectations.  Finally,  DOD 
stated  that  because  the  effectiveness  of  annual  continuation  training  is 
identified  at  the  local  level,  only  the  accomplishment  of  training  needs  to 
be  collected  and  reported.  We  agree  that  squadron  commanders  should 
have  the  flexibility  to  inform  individual  aircrew  training  needs.  However, 
establishing  a  baseline  of  desired  learning  objectives  to  meet  Air  Combat 
Command’s  training  expectations  for  the  Ready  Aircrew  Program  will 
provide  squadron  commanders  information  on  what  features  a  training 
event  should  include  and  help  ensure  that  aircrews  are  receiving 
consistent  and  comparable  training  across  the  force.  Further,  while  the 
Ready  Aircrew  Program  tasking  memorandums  identify  desired  adversary 
air  ratios,  the  memorandums  do  not  identify  any  other  training  support 
elements  that  are  needed  to  accomplish  training  expectations.  For 
example,  the  tasking  memorandums  do  not  provide  information  on  the 
training  range  capabilities  needed  to  meet  expectations,  such  as  threat 
replicators  and  targets.  As  noted  in  our  report,  a  range  of  factors,  in 
addition  to  the  availability  of  sufficient  ratios  of  adversary  air,  have  limited 
the  ability  of  combat  fighter  squadrons  to  complete  annual  training 
requirements.  The  lack  of  this  information  can  affect  the  Air  Force’s  ability 
to  make  informed  resource  decisions  to  improve  its  training  infrastructure. 
Finally,  our  report  also  noted  that  Air  Combat  Command  guidance  states 
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that  it  expects  to  see  data  that  documents  the  underlying  challenges  to 
providing  an  effective  training  program.  DOD’s  comment  that  only  the 
accomplishment  of  training  needs  to  be  collected  and  reported  contradicts 
this  guidance  as  well  as  the  Air  Force’s  stated  goal  of  moving  to  a  training 
model  based  on  the  effectiveness  of  training,  using  quantifiable  levels  of 
performance,  rather  than  the  current  model  focused  on  the  frequency  of 
training.  Without  developing  a  process  to  collect  data  and  measure  the 
effectiveness  of  training  against  established  expectations  that  includes 
information  on  desired  learning  objectives  and  training  support  elements, 
the  Air  Force  will  not  have  the  information  required  to  achieve  this  goal. 
Therefore,  we  believe  the  recommendation  remains  valid. 

In  concurring  with  the  third  recommendation,  DOD  stated  that  the  Air 
Force  is  beginning  to  take  a  holistic  view  of  operational  training  and  the 
Air  Force  has  stood  up  an  Operational  Training  Division  to  better  focus 
control  over  the  operational  training  enterprise.  One  responsibility  of  the 
division  will  be  to  update  the  United  States  Air  Force  Live  Virtual 
Constructive  Operational  Training  Flight  Plan  with  all  of  the  desirable 
characteristics  of  a  comprehensive  strategy.  The  Air  Force  expects  to 
complete  this  update  by  September  2017.  By  updating  this  plan  to  include 
the  desirable  characteristics  of  a  comprehensive  strategy,  such  as  a  risk- 
based  investment  strategy  that  identifies  and  prioritizes  capability  needs 
and  a  time  line  for  addressing  them,  the  Air  Force  would  improve  its 
ability  to  develop  the  capabilities  required  to  achieve  its  virtual  training 
needs. 


We  are  sending  copies  of  this  report  to  the  appropriate  congressional 
committees,  the  Secretary  of  Defense,  the  Under  Secretary  of  Defense 
for  Personnel  and  Readiness,  and  the  Secretary  of  the  Air  Force.  In 
addition,  the  report  is  available  at  no  charge  on  the  GAO  website  at 
http://www.gao.gov. 

If  you  or  your  staff  have  questions  about  this  report,  please  contact  me  at 
(202)  512-5431  or  russellc@gao.gov.  Contact  points  for  our  Offices  of 
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Congressional  Relations  and  Public  Affairs  may  be  found  on  the  last  page 
of  this  report.  GAO  staff  who  made  key  contributions  to  this  report  are 
listed  in  appendix  III. 


Cary  Russell 

Director,  Defense  Capabilities  and  Management 
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List  of  Committees 


The  Honorable  John  McCain 
Chairman 

The  Honorable  Jack  Reed 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Thad  Cochran 
Chairman 

The  Honorable  Richard  J.  Durbin 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Mac  Thornberry 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Rodney  Frelinghuysen 
Chairman 

The  Honorable  Pete  Visclosky 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Appendix  I:  Objectives,  Scope,  and 
Methodology 


The  objectives  of  this  report  are  to  determine  the  extent  to  which  the  Air 
Force  has  (1)  determined  requirements  to  train  its  combat  aircrews  for  the 
full  range  of  core  missions,  (2)  met  annual  training  requirements  for 
combat  fighter  squadrons  across  the  full  range  of  core  missions  and 
evaluated  the  effectiveness  of  this  training,  and  (3)  established  virtual 
training  plans  that  include  desirable  characteristics  of  a  comprehensive 
strategy. 

This  report  is  a  public  version  of  the  prior  sensitive  report  that  we  issued 
in  August  2016. 1  The  Department  of  Defense  (DOD)  deemed  some  of  the 
information  in  the  prior  report  as  For  Official  Use  Only  (FOUO),  which 
must  be  protected  from  public  disclosure.  Therefore,  this  report  omits 
FOUO  information  and  data  on  some  of  the  Air  Force’s  training  priorities, 
completion  of  annual  training  requirements  for  active-duty  fighter 
squadrons,  and  aircraft  maintenance  generation  capabilities.  Although  the 
information  provided  in  this  report  is  more  limited  in  scope,  it  addresses 
the  same  objectives  as  the  sensitive  report.  Also,  the  methodology  used 
for  both  reports  is  the  same. 

We  focused  our  review  on  the  annual  continuation  training  requirements 
for  aircrews  assigned  to  aircraft  within  the  active  component  of  the 
combat  air  forces.2  We  focused  on  Air  Combat  Command  as  the  lead 
major  command  for  all  combat  air  forces,  which  has  responsibility  for 
developing  and  managing  annual  continuation  training  requirements 
through  the  Ready  Aircrew  Program.  The  aircraft  included  in  our  review 
were  the  A-10,  F-15,  F-16,  F-22,  and  F-35.  We  selected  a  non- 
generalizable  sample  of  units  to  speak  with  based  on  our  analysis  of 
annual  training  completion  data  and  the  range  of  units’  core  missions. 
Based  on  this  analysis,  we  conducted  site  visits  with  F-22  units  at  Langley 
Air  Force  Base,  F-16  Block  40  and  F-35  units  at  Hill  Air  Force  Base,  F- 
15E  units  at  Seymour  Johnson  Air  Force  Base,  and  F-16  Block  50  units  at 
Shaw  Air  Force  Base. 


^AO,  Air  Force  Training:  Further  Analysis  and  Planning  Needed  to  Improve 
Effectiveness,  GAO-16-635SU  (Washington,  D.C.:  Aug.  16,  2016). 

2ln  this  report,  we  refer  to  continuation  training  as  annual  training  requirements.  The 
Ready  Aircrew  Program  establishes  the  annual  training  requirements  aircrews  must 
accomplish  to  sustain  combat  mission  readiness. 
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Appendix  I:  Objectives,  Scope,  and 
Methodology 


To  determine  the  extent  to  which  the  Air  Force  has  determined 
requirements  to  train  its  combat  aircrews  for  the  full  range  of  core 
missions,3  we  analyzed  Air  Force  documents  on  training  to  identify  the  Air 
Force’s  current  process  for  developing  training  requirements,  including 
the  Air  Force  Instruction  11-2,  Volume  1  for  each  aircraft  in  our  review. 

We  interviewed  officials  involved  in  the  development  of  the  annual 
training  requirements,  including  officials  from  Air  Combat  Command,  U.S. 
Air  Forces  Europe,  Pacific  Air  Forces,  and  U.S.  Air  Forces  Central 
Command.  We  also  interviewed  wing  and  squadron  commanders  at 
selected  units,  as  described  above,  to  obtain  an  operational-level 
perspective  on  the  training  requirements  needed  to  meet  the  full  range  of 
core  missions.  We  reviewed  the  annual  training  requirements  established 
for  each  aircraft  for  fiscal  years  2012  through  2016  to  determine  how  the 
Air  Force  establishes  the  experienced  and  inexperienced  aircrew 
designations  and  how  sortie  and  simulator  mission  requirements  have 
changed  over  time.  We  evaluated  these  processes  against  Department  of 
Defense  (DOD)  and  Air  Force  guidance  that  prioritize  and  establish 
training  requirements  and  against  the  leading  practices  we  have  identified 
for  managing  strategic  training.4 

To  determine  the  extent  to  which  the  Air  Force  has  met  annual  training 
requirements  for  combat  fighter  squadrons  across  the  full  range  of  core 
missions  and  evaluated  the  effectiveness  of  this  training,  we  analyzed 
training  completion  data  from  fiscal  years  2012  through  2015  to  identify 
core  missions  for  which  aircrews  were  or  were  not  fully  trained.  We 
performed  data  reliability  procedures  on  the  information  included  in 
Ready  Aircrew  Program  completion  reports  by  comparing  that  information 


3For  the  purposes  of  this  report,  we  use  the  phrase  “full  range  of  core  missions”  to 
describe  the  primary  and  secondary  missions  as  established  in  each  aircraft’s  Designed 
Operational  Capability  statement,  which  summarizes  specific  capabilities  that  an  Air  Force 
unit  can  be  called  on  to  provide. 

4GAO,  Human  Capital:  A  Guide  for  Assessing  Strategic  Training  and  Development  Efforts 
in  the  Federal  Government ,  GAO-04-546G  (Washington  D.C.:  Mar.  1, 2004).  This  guide 
introduces  a  framework,  consisting  of  a  set  of  principles  and  key  questions  that  federal 
agencies  can  use  to  ensure  that  their  training  and  development  investments  are  targeted 
strategically.  Information  in  this  guide  was  developed  through  consultations  with 
government  officials  and  experts  in  the  private  sector,  academia,  and  nonprofit 
organizations;  examinations  of  laws  and  regulations  related  to  training  and  development  in 
the  federal  government;  and  a  review  of  the  sizeable  body  of  literature  on  training  and 
development  issues,  including  previous  GAO  products  on  a  range  of  human  capital  topics. 
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Appendix  I:  Objectives,  Scope,  and 
Methodology 


against  related  documentation  and  interviewing  knowledgeable  officials 
on  controls  over  the  reporting  systems.  We  determined  that  the  data 
presented  in  our  findings  were  sufficiently  reliable  for  the  purposes  of  this 
report.  We  reviewed  Air  Force  documentation  that  describes  training 
challenges  and  interviewed  Air  Force  officials,  including  wing  and 
squadron  commanders  at  selected  units,  as  described  above,  and 
officials  at  Air  Combat  Command’s  two  major  range  and  test  facility 
bases — the  Nevada  Test  and  Training  Range  and  the  Utah  Test  and 
Training  Range — to  discuss  any  factors  that  limit  the  ability  of  aircrews  to 
complete  training  for  the  full  range  of  core  missions.  We  evaluated  the  Air 
Force’s  process  for  assessing  the  effectiveness  of  its  annual  training 
against  leading  practices  we  identified  for  managing  strategic  training  and 
against  the  Ready  Aircrew  Program  tasking  memorandums,  which  specify 
annual  continuation  training  for  personnel  assigned  to  combat  units.5 

To  determine  the  extent  to  which  the  Air  Force  has  established  virtual 
training  plans  that  include  desirable  characteristics  of  a  comprehensive 
strategy,6  we  reviewed  planning  documents  established  by  the  Air  Force 
to  guide  its  virtual  training  efforts  and  evaluated  these  actions  against  our 
prior  work  on  desirable  characteristics  of  national  strategies.  Given  that 
there  is  no  established  set  of  requirements  for  strategies,  we  relied  on  our 
assessments  of  national  strategies  and  the  criteria  that  were  applied  to 
assess  these  strategies.7  We  identified  six  desirable  characteristics  that 
national  or  department-wide  strategies  should  contain.  We  assessed 
these  criteria  against  the  two  existing  Air  Force  planning  documents  we 


5GAO-04-546G. 

According  to  Air  Force  documents,  the  live  (L)  environment  is  defined  as  real  people 
operating  real  weapons  systems,  the  virtual  (V)  environment  is  defined  as  real  people 
operating  simulated  systems,  and  the  constructive  (C)  environment  is  identified  as 
software  models  and  code  that  are  used  to  improve  training  scenarios  with  computer¬ 
generated  entities — such  as  terrain,  threats,  aircraft,  people,  and  vehicles,  among  others. 
For  the  purposes  of  this  report,  we  refer  to  training  that  includes  a  simulator  as  virtual 
training. 

7We  assessed  planning  documents  established  by  the  Air  Force  to  guide  its  virtual  training 
efforts  against  criteria  we  developed  for  national  strategies.  These  criteria  are  cited  in 
GAO,  Combating  Terrorism:  Evaluation  of  Selected  Characteristics  in  National  Strategies 
Related  to  Terrorism,  GAO-04-408T  (Washington,  D.C.:  Feb.  3,  2004).  Given  the  size  and 
complexity  of  the  Air  Force’s  virtual  training  efforts,  we  concluded  that  a  comprehensive 
strategy  to  guide  these  efforts  would  be  similar  in  scope  and  would  need  to  include 
characteristics  similar  to  those  of  a  national  strategy. 
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found  most  relevant  to  the  combat  air  forces — United  States  Air  Force 
Live  Virtual  Constructive  Operational  Training  Flight  Plan  and  the  Combat 
Air  Force  Live,  Virtual,  Constructive  Vision — to  determine  the  extent  to 
which  these  documents  contain  the  elements  of  a  comprehensive 
strategy.  We  determined  whether  these  actions  addressed,  partially 
addressed,  or  did  not  address  the  desirable  characteristics  of  strategy 
documents,  based  on  our  assessment  of  whether  the  planning 
documents  explicitly  cited  all  elements  of  a  characteristic  and  the  level  of 
specificity  and  detail  included. 

We  conducted  this  performance  audit  from  June  2015  to  September  2016 
in  accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 

We  interviewed  officials  and,  where  appropriate,  obtained  documentation 
at  the  following  locations: 

•  Office  of  the  Secretary  of  Defense,  Office  of  the  Under  Secretary  of 
Defense  for  Personnel  and  Readiness 

•  Headquarters  Air  Force,  Current  Operations 

•  Headquarters  Air  Force,  Readiness  and  Exercises 

•  Headquarters  Air  Force,  Live,  Virtual,  and  Constructive/Modeling  and 
Simulation 

•  Air  Force  Agency  for  Modeling  and  Simulation 

•  Air  Combat  Command,  Flight  Operations 

•  Air  Combat  Command,  Airspace,  Ranges,  and  Airfield  Operations 

•  Air  Combat  Command,  Operations 

•  Air  Combat  Command,  Resources  and  Budget 

•  U.S.  Air  Force  Warfare  Center 

•  Nevada  Test  and  Training  Range 

•  Langley  Air  Force  Base,  1st  Fighter  Wing 

•  Langley  Air  Force  Base,  27th  Fighter  Squadron 

•  Langley  Air  Force  Base,  94th  Fighter  Squadron 


Page  42 


GAO-16-864  Air  Force  Training 


Appendix  I:  Objectives,  Scope,  and 
Methodology 


•  Hill  Air  Force  Base,  388th  Fighter  Wing 

•  Hill  Air  Force  Base,  4th  Fighter  Squadron 

•  Hill  Air  Force  Base,  34th  Fighter  Squadron 

•  Utah  Test  and  Training  Range 

•  Shaw  Air  Force  Base,  20th  Fighter  Wing 

•  Shaw  Air  Force  Base,  77th  Fighter  Squadron 

•  Shaw  Air  Force  Base,  79th  Fighter  Squadron 

•  Seymour  Johnson  Air  Force  Base,  4th  Fighter  Wing 

•  Seymour  Johnson  Air  Force  Base,  335th  Fighter  Squadron 

•  Seymour  Johnson  Air  Force  Base,  336th  Fighter  Squadron 

•  Pacific  Air  Forces,  Exercises  and  Readiness 

•  U.S.  Air  Forces  Europe,  Training,  Readiness,  and  Exercises 

•  U.S.  Air  Forces  Central  Command,  Air  Operations  and  Training 
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OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 

4000  DEFENSE  PENTAGON 
WASHINGTON,  D.C.  20301-4000 


READINESS  JUL  2  7  2016 

Mr.  Cary  Russell 

Director,  Defense  Capabilities  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  NW 
Washington  DC  20548 

Dear  Mr.  Russell: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report 
GAO-16-635,  “AIR  FORCE  TRAINING:  Further  Analysis  and  Planning  Needed  to  Improve 
Effectiveness,”  dated  May  25,  2016  (GAO  Code  100139). 

The  Department  appreciates  the  opportunity  to  comment  on  this  draft.  We  see  value  in 
the  GAO’s  review  but  we  non  concur  with  Recommendations  1  and  2,  and  concur  with 
Recommendation  3.  The  Department’s  comments  on  the  GAO  draft  recommendations  are 
enclosed. 

The  Defense  Office  of  Prepublication  and  Security  Review  found  that  portions  of  the 
draft  GAO  report  contain  For  Official  Use  Only  (FOUO)  material  that  is  not  publicly  releasable. 
The  detailed  results  of  that  security  review  have  been  provided  to  you  under  separate  cover. 

My  point  of  contact  is  Frank  C.  DiGiovanni  who  can  be  reached  at 
frank.c.digiovanni.civ@mail.mil  or  703-695-2618. 


Sincerely, 


S’ft"'  Diana  C.  Banks 

Deputy  Assistant  Secretary  of  Defense 
(Force  Education  and  Training) 


Attachments: 
As  stated 
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GAO  DRAFT  REPORT  DATED  MAY  25, 2016 
GAO-16-635  (GAO  CODE  100139) 

“AIR  FORCE  TRAINING:  FURTHER  ANALYSIS  AND  PLANNING  NEEDED  TO 
IMPROVE  EFFECTIVENESS” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATION 


RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense  direct  the 
Secretary  of  the  Air  Force  to  comprehensively  reassess  the  assumptions  underlying  its  annual 
training  requirements — including,  but  not  limited  to,  the  total  annual  training  requirements  by 
aircraft,  the  criteria  for  designating  aircrews  as  experienced  or  inexperienced,  and  the  mix 
between  live  and  simulator  training — and  make  any  appropriate  adjustments  in  future  training 
plans. 

DoD  RESPONSE:  Non  concur.  The  Department  is  has  already  begun  a  comprehensive 
reassessment  of  the  assumptions  underlying  annual  training  requirements  and  those  results  will 
be  used  to  influence  future  training  plans.  Regarding  the  “total  annual  training  requirements  by 
aircraft”  and  “the  mix  between  live  and  simulator  training,”  Air  Combat  Command  (ACC)  has 
directed  the  funding  and  execution  of  a  comprehensive  study  to  reassess  the  full  spectrum  of 
fighter  Ready  Aircrew  Program  (RAP)  requirements.  A  Request  for  Proposal  is  being  finalized 
with  an  anticipated  contract  start  date  in  September  2016.  It  is  important  to  note  that  RAP 
currently  establishes  the  minimum  level  of  training  for  a  Combat  Mission  Ready  aircrew  but  is 
often  interpreted  as  a  maximum.  ACC  anticipates  study  results  will  be  available  about  12 
months  after  contract  award  that  will  provide  a  range  of  readiness  levels  associated  with  varying 
training  levels.  Regarding  “the  criteria  for  designating  aircrews  as  experienced  or 
inexperienced,”  ACC  directed  a  study  of  all  factors  that  contribute  to  aircrew  experiencing  in  the 
F-22  and  the  completed  study  resulted  in  substantive  changes  for  F-22  experiencing  criteria. 
ACC  has  commissioned  a  follow-on  study  to  include  analysis  of  the  experiencing  criteria  of  all 
4th  Generation  fighters  (A-10,  F-15C,  F-15E,  and  F-16).  This  follow-on  study  is  expected  be 
completed  by  30  Sep  2017. 


RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense  direct  the 
Secretary  of  the  Air  Force  to: 

*  establish  desired  learning  objectives  and  training  support  elements  needed  to 
accomplish  the  training  expectations  in  its  annual  Ready  Aircrew  Program  tasking 
memorandums;  and 

*  develop  a  process  to  collect  data  to  assess  the  effectiveness  of  annual  training  against 
these  features. 
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DoD  RESPONSE:  Non  concur.  The  Department’s  position  is  that  aircrew  training  differs 
significantly  from  other  syllabus-directed  courses  of  instruction.  Ready  Aircrew  Program  (RAP) 
training  methods  and  reporting  is  a  robust  set  of  institutionalized  processes  that  meet  or  exceed 
all  regulatory  requirements  as  well  as  the  needs  of  USAF  leadership.  Regarding  “establish 
desired  learning  objectives,”  GAO  frequently  refers  to  its  previously  published  report  Human 
Capital:  A  Guide  for  Assessing  Strategic  Training  and  Development  Efforts  in  the  Federal 
Government  in  the  analysis  of  the  USAF  fighter  continuation  training  program  (i.e.,  RAP); 
however,  RAP  differs  significantly  from  other  syllabus-directed  courses  of  instruction.  While 
fighter  aircrew  do  accomplish  formalized  upgrade  training,  which  adheres  to  principles  and 
guidelines  described  in  the  GAO  guide,  RAP  directs  training  that  is  not  governed  by  a  syllabus. 
Desired  learning  objectives  for  RAP  sorties  are  set  by  the  flight  lead  based  on  the  training  needs 
and  experience  levels  of  the  flight  members.  The  fighter  RAP  Tasking  Memoranda  (RTM)  for 
all  Mission  Design  Series  (MDS),  direct  that  Squadron  Commanders  should  “...develop  unit 
training  programs  that  focus  on  their  primary  and  secondary  missions.”  Further,  the  RTMs  state: 
“Per  AFI  1 1-2(MDS),  Volume  1,  an  effective  RAP  mission  requires  accomplishing  a  complete 
tactical  mission  scenario  or  a  basic  skills  mission.  In  order  to  be  effective,  each  mission  must 
successfully  complete  a  sufficient  number  of  events  applicable  to  that  mission  type,  according  to 
the  Squadron  Commander.”  Finally,  AFI  1 1-2(MDS),  Volume  3  provides  briefing  guides  that 
direct  the  flight  lead  to  set  Mission  Objectives.  This  guidance  provides  Fligher  Headquarters 
direction  at  the  appropriate  level  of  fidelity  while  providing  local  Commanders  and  leaders  the 
flexibility  to  set  detailed  learning  objectives  for  individual  sorties,  which  in  turn,  allows  them  to 
maximize  the  training  value  and  efficiency  of  each  sortie.  Employment  of  fighter  aircraft  is  a 
fleeting  skill  set  that  requires  the  adjustment  of  training  objectives  on  a  day-to-day  and  individual 
basis.  Regarding  “training  support  elements,”  RTMs  for  the  F-15C,  F-15E,  F-16,  F-22,  and  F-35 
contain  Table  A1  that  lists  Desired  Adversary  Ratios.  This  table  serves  to  list  “training  support 
elements”  needed  to  accomplish  the  training  expectations  and  is  derived  from  Combat- 
Numbered  Air  Forces  expectations.  Further,  ACC  is  partnering  with  our  sister  services  to 
maximize  joint  force  training  opportunities  in  Close  Air  Support,  Maritime  Air  Support,  Combat 
Search  and  Rescue,  and  Forward  Air  Controller  (Airborne)  both  as  the  force  being  trained  and 
the  training  support  element.  Regarding  “develop  a  process  to  collect  data  to  assess  the 
effectiveness  of  annual  training  against  these  features,”  once  it  is  understood  that  desired 
learning  objectives  and,  derivatively,  measures  of  effectiveness  for  RAP  continuation  training 
are  set  at  the  local  level,  it  becomes  clear  that  accomplishment  and  not  effectiveness  is  the 
appropriate  metric  to  be  reported  to  the  Major  Command. 

RECOMMENDATION  3 :  The  GAO  recommends  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Air  Force  to  continue  to  refine  its  planning  for  virtual  training  to 
incorporate  the  desirable  characteristics  of  a  comprehensive  strategy,  including  developing  a 
risk-based  investment  strategy  that  identifies  and  prioritizes  capability  needs  and  includes  a 
time  line  for  addressing  them. 

DoD  RESPONSE:  Concur.  The  Department  is  taking  a  holistic  view  of  the  Operational 
Training  it  accomplishes  in  both  Live  and  Virtual  (Synthetic)  environments.  Additionally, 
across  the  Department,  due  primarily  to  the  emergence  of  5th  generation  weapon  system 
capabilities.  Blended  Training  (live  training  enhanced  with  synthetic  entities)  is  getting  more 
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attention  and  development  planning  has  already  begun.  Consequently,  the  Air  Force  Director 
of  Current  Operations  (AF/A30)  recently  stood  up  an  Operational  Training  Division 
(AF/A30T)  to  better  focus  his  span  of  control  over  the  Operational  Training  enterprise.  One 
of  the  first  responsibilities  of  the  new  Operational  Training  Division  is  to  update  the 
UNITED  STATES  AIR  FORCE  LIVE  VIRTUAL  CONSTRUCTIVE  OPERATIONAL 
TRAINING  FLIGHT  PLAN  (22  Feb  2013).  This  updated  document,  expected  by 
September,  2017,  will  incorporate  all  the  desirable  characteristics  of  a  comprehensive 
strategy  and  provide  a  vision  for  how  the  Air  F orce  intends  to  optimize  its  operational 
training. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and  investigative 
arm  of  Congress,  exists  to  support  Congress  in  meeting  its  constitutional 
responsibilities  and  to  help  improve  the  performance  and  accountability  of  the 
federal  government  for  the  American  people.  GAO  examines  the  use  of  public 
funds;  evaluates  federal  programs  and  policies;  and  provides  analyses, 
recommendations,  and  other  assistance  to  help  Congress  make  informed 
oversight,  policy,  and  funding  decisions.  GAO’s  commitment  to  good  government 
is  reflected  in  its  core  values  of  accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost  is 
through  GAO’s  website  (http://www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  website  newly  released  reports,  testimony,  and  correspondence.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products,  go  to  http://www.gao.gov 
and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of  production  and 
distribution  and  depends  on  the  number  of  pages  in  the  publication  and  whether 
the  publication  is  printed  in  color  or  black  and  white.  Pricing  and  ordering 
information  is  posted  on  GAO’s  website,  http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 
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Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512-4400, 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125, 
Washington,  DC  20548 
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